In working towards more complete principles for ME it is necessary to place this work in the context of similar work reported in the literature. Accordingly, in the following subsections we describe the currently prevailing view of “ideal” ME in terms of its process, criteria, and deliverables. This allows us to analyze alternative ME approaches and describe their underlying assumptions, as well as their weaknesses and strengths. Moreover, and most importantly, the view of current ME principles allows us to describe what are their differences in relation to our focus on ME, namely to engineer modeling techniques for tools.

The general structure of a ME process (cf. Smolander et al. 1990, Tolvanen and Lyytinen 1993, Brinkkemper 1996, Cronholm and Goldkuhl 1994, Grundy and Venable 1996, Harmsen 1997) is illustrated in Figure 3-2. The model follows the notation of data flow diagrams (Yourdon 1989a) in which processes are circles, external entities are rectangles, and data stores are rounded rectangles. The arrows describe data flows between processes, externals and stores.

FIGURE 3-2 A data flow diagram specifying ME process.

In the following we outline the ME process by describing each step, namely method selection, method construction and tool adaptation. These are described as processes in the figure. It must be noticed that the figure does not include all steps of local method development (cf. Figure 1-1), such as method introduction, use, or collection of experiences, since the ME literature does not provide any systematic principles for these, although the tasks are usually acknowledged.

In the method selection process the ISD environment is analyzed according to ME criteria. The criteria for methods can be divided into situation-independent and situation-dependent parts. The former criteria are considered desirable for most methods regardless of the situation for which they are developed. Examples of these universal criteria are: easiness to learn, simplicity of use, good support for communication between stakeholders and good support for transitions between different tasks or phases of ISD. These criteria cover more than one type of method knowledge, but can also be specific only to certain types of method knowledge. In our case of constructing modeling techniques examples of general criteria include readability and easy to use.

The latter type of criteria are relevant when we want to increase the applicability of a method for a given situation. Jarke et al. (1998) call these method adaptation criteria, and they are of primary interest for incremental ME. They include classifications of relevant aspects of methods which should be considered to satisfy the objectives for the method. For example, in carrying out IS planning, the degree of centralization of the target organization is suggested as one criteria (Sullivan 1985). If the organization is centralized, IS planning can be performed better with BSP (IBM 1984), whereas de-centralized organizations can be analyzed better with CSF (Rockart 1979). Among the ME criteria we can distinguish between criteria which relate to contingencies, development problems, and stakeholders’ values. These criteria are reviewed in more detail in Section 3.2.3.

The selected method (or methods) which meet the ME criteria are constructed, possibly with new method components. This means combining method knowledge from different methods as well as from different types of method knowledge. By focusing on method components (or fragments, Harmsen 1997), and therefore introducing smaller changes, methods can be maintained or even integrated (e.g. Kronlöf 1993) if required. For example, if the programming language changes from C++ to Smalltalk, the method knowledge is modified slightly: it is no longer permissible to use multiple inheritance. Since all criteria are not necessarily met with existing methods, new method knowledge needs to be defined, and some of the method components may need to be removed. The new method configuration is stored into a repository for future selections.

Finally, the method constructed needs to be adapted into a CASE tool. Generally speaking tool adaptation deals with customizing or building a tool for the method, or choosing a set of tools which cover all the method knowledge (for selection strategies see Bubenko 1988). If this adaptation is not carried out then the contribution of the method construction is limited, because a tool could ensure that the method is used as intended. ME without tool adaptation would be the same as developing IS specifications without implementing associated computer-based support. Method introduction also involves non-computer supported parts, such as production of manuals, tutorials, etc., which are not included into the adapted tool. Basically, tool adaptation includes building method-tool companionship, namely the support for abstractions, model checking, form conversion and review (cf. Section 2.3.2). This adaptation requires that the constructed method is modeled with a modeling language offered by the customizable CASE tool. If the CAME tool provides translation of the metamodels into a CASE tool (e.g. Rossi et al. 1992), or method use can be tested in the same tool where the tool adaptation is performed (e.g. Kelly 1997), the tool adaptation becomes easier and faster. Hence, an outcome of a successful ME process is a fully functional computer-based IS for ISD. This defines what developers can store into the repository of an ISD tool, how system descriptions can be represented, retrieved, checked, transformed, and how descriptions are managed.

Ideally speaking all sorts of method knowledge and their relationships can be subject to ME: from the underlying conceptual structure to the assumptions and value-orientations of a method. Practically speaking, methods usually only address a few types of method knowledge (Jarke et al. 1998), and methods can be modified by changing only one or a few types of method knowledge (e.g. Kronlöf 1993).

Although the types of method knowledge are related, each type can be viewed independently and represented using a number of alternative representation schemes and mechanisms resulting in different kind of metamodels. Clearly, no method construction is possible without some sort of (explicit or implicit) metamodeling. Thus behind all approaches there are some metamodeling formalisms (cf. Section 3.3). In the following a set of ME approaches are analyzed based on their focus on different types of method knowledge. These ME approaches are taken from a survey of ME research (Tolvanen et al. 1996) and the approaches are summarized in Table 3-1. It must be noticed that a single approach may belong to more than one category, as they typically cover at least modeling of conceptual structures, even if only as the foundation for the modeling notation, procedural guidelines, participation or values.

Because all methods are based on some concepts, ME approaches address the conceptual structure, at least those concepts related to the other type of method knowledge addressed. There are, however, ME approaches which focus mainly on conceptual structure, such as Mercurio et al. (1990), Essink (1988), Olle et al. (1991), and Heym and Österle (1992). In general, these frameworks present conceptual structures or suggest reference models of methods. Hence, here the objective of ME is to identify and establish relevant concepts of ISD and include them in the conceptual structure of the method. A short example of a conceptual structure is now in order. For example, the conceptual structure of most object-oriented methods includes the concept of inheritance, its relation to other concepts, and possibly constraints, such as single inheritance or recursivity. Although this conceptual structure also includes relationships to notations, or to processes, they are often too general to define methods in such a detailed and formal manner that computer support can be built based solely on such a conceptual structure.

TABLE 3-1 Method engineering approaches and types of method knowledge

The most studied ME approaches are those operating at the notational level, but related to conceptual structures (Tolvanen et al. 1996). These approaches define modeling techniques (e.g. Teichroew et al. 1980, Sorenson et al. 1988, Welke 1988, Smolander 1991, Saeki and Wenyin 1994, Venable 1993). Some of them (e.g. Sorenson et al. 1988, Welke 1988, Smolander 1991, Kelly et al. 1996) are also used as a conceptual schema for modeling tools. Notation-based ME is more concise than those based on conceptual structures: it focuses only on the concepts related to modeling. For example, it defines how the concept of inheritance is represented, related to the representations of other concepts, and how the constraints of the inheritance are supported in a modeling technique. Hence, while focusing on modeling techniques they address the conceptual-representational dimension (Smolander et al. 1990) by defining how concepts are represented. This is typically achieved by relating conceptual structures to their representation definitions (e.g. Smolander et al. 1991, Wijers 1991). For example, Smolander et al. (1991) link the conceptual content of a method into a graphical, diagram-oriented representation. In Kelly (1994) this approach is extended into a matrix representation. As a result, these method definitions can be applied at the ISD level as a modeling technique. The metamodels representing modeling techniques are often characterized as meta-data models (Brinkkemper 1990). This thesis investigates notation-based ME, and therefore we describe in Section 3.3 only those metamodels and metamodeling languages which support a notation-based approach.

ME approaches that concentrate on the processes of method use are less developed than those that concentrate on the conceptual structure and notations of methods[12] (Tolvanen et al. 1996). They can be divided into those that specify the relationships among the modeling tasks (e.g. Wijers 1991, Hofstede and Nieuwland 1993, Rolland and Prakash 1996, Jarke et al. 1994) and those that emphasize the specification of modeling products and the tasks needed to make them change (e.g. Marttiin 1994). The former can be used to describe, for example, when and how inheritance structures are identified, and the latter also in which technique it is represented. These processes are represented in process or meta-activity models (Brinkkemper 1990). A process model is always related to the conceptual structure of a method, but it can also be related to the notation based metamodels, as in the latter example. The manipulation of models is always dictated by tasks and decisions, whether or not these are defined in an explicit process model. In this thesis, we do not consider the process-based approaches to modeling techniques. The strategies of integrating a meta-data model and a process model are discussed in Wijers (1991), Marttiin et al. (1995), Tolvanen et al. (1993), and Kinnunen and Leppänen (1994). Classifications of process models can be found from Dowson (1987) and surveys from Curtis et al. (1992) and Finkelstein et al. (1994).

Other types of method knowledge are more poorly addressed in the ME literature. One reason is the absence of such descriptions in the method books. At the level of ISD participation, ME approaches define the stakeholders involved and the organizational structures related to method use. The metamodel of Tolvanen et al. (1993) includes agent models which specify the activities performed and the agents involved. The Method Engineering Language (MEL) of Harmsen (1997) allows the entry of different roles, such as responsibility, for method components. Therefore, a participation model of ISD defines, for example, who is responsible for finding and creating inheritance hierarchies with class diagrams.

At the level of modeling decisions a variety of design rationale approaches are proposed. These aim to record the design decisions made based on a predefined schema (Ramesh and Edwards 1993). Originally the decision-oriented models focused on decisions behind designs not behind methods, e.g. why an inheritance between two classes is defined as virtual. Design rationale can be also modeled in two other ways which are beneficial to ME: decisions related to method use and decisions related to method construction. An example of the former could be an IS developer’s justification why a concept of virtuality is used in inheritance structures. Approaches of this type focus on decisions related to the ISD process. The proposal of Rolland et al. (1995) focuses on the specification of successive transformations of the modeling product, from the viewpoint of the consequences of decisions. Jarke et al. (1994) propose a traceability model for tracing processes defined in a guidance model. An example of the latter would be a method engineer’s justification of why a metamodel includes a concept of inheritance. Oinas-Kukkonen (1996) proposes metamodel-related rationale, but does not explain its use in more detail. For example, to which types of method knowledge should the decisions on method use be related? Should it be used for recording why a certain concept was used, why it was represented in a specific diagram, or why it was specified in a certain phase of ISD, etc.

Finally, to our knowledge only Kumar and Welke (1984, 1992) have directly focused in the ME literature on the development objectives and values underlying methods. They proposed an ISD-Personal Value Questionnaire (ISD-PVQ), consisting of 86 value concepts, with which a method stakeholder’s values can be collected and their relevance can be assessed within three different groups, addressing technical values, such as timeliness of information; economical values, such as ISD costs; or socio-political-psychological values, such as system responsiveness to people.

Although all types of method knowledge can be modeled with ME approaches, it does not mean that all types of method knowledge are modeled fully. First, there are many activities like brainstorming sessions, meetings, etc. which are not even considered to be modeled with current ME approaches. Second, part of the knowledge applied in carrying out ISD is tacit and therefore not expressible. As a consequence, aspects of method knowledge which can not be made explicit can not be improved through method engineering principles, nor supported by ISD tools. Finally, it must also be noticed that there are dependencies between different types of method knowledge (cf. Figure 2-2). Therefore, in local method development it is not meaningful to focus solely on one or a few types of method knowledge. For example, the balancing rules (Yourdon 1989a) necessitate that a notation-related metamodel can recognize mappings between data stores and entities, although these mappings do not have a notational counterpart. Hence, it must be noticed that metamodeling is difficult, if not impossible, to perform meaningfully without considering other types of method knowledge.

ME approaches must be also characterized according to the driving factors or criteria used to engineer methods: it is not useful to perceive and model method knowledge if you do not know how it should be analyzed, constructed and maintained. Accordingly, most important to the success of ME is not how well a method can be represented, but how the applicability of a method is improved.

In the following, ME approaches are analyzed based on which kind of criteria they apply to achieve the methodical requirements of ISD. In principle, these approaches fall into three categories, namely those describing the method use environments, also known as contingency frameworks; those emphasizing the importance of problems at hand to be solved by the method; and those focusing directly on method users’ requirements. These approaches are summarized in Table 3-2.

TABLE 3-2 Method engineering approaches and criteria.

It must be noticed that the three categories are not necessarily the same approaches as those addressing types of method knowledge. Many of the approaches discussed earlier focus mostly on metamodeling but do not explicitly describe what should be done with the metamodels. Compared to research on metamodeling languages, the criteria for engineering methods seem to be less studied. It must also be noticed that they can overlap. For example, a contingency framework can also include some criteria related to different type of problem situations or aspects of the stakeholders’ values. Moreover, the same criteria can be recognized with more than one type of method engineering approach.

Each approach focuses on different aspects of ME, and they are therefore limited to some extent. Below, we discuss their weaknesses and strengths as principles for carrying out ME.

The majority of ME approaches apply contingency frameworks (cf. Section 1.4.2) to characterize an ISD environment and to find situational requirements for methods. This characterization is performed through an analysis of the project’s context (Slooten and Hodes 1996), its environment (Harmsen 1997, Harmsen et al. 1994b) or the profile of the situation (Vlasblom et al. 1995). The content and objective of these approaches, however, are the same. The applicability of a method is understood as the closest possible relationship between the characteristics of the ISD situation and the characteristics of the method.

The approaches analyzed use either an external contingency framework (e.g. van Slooten and Hodes 1996), or relate some situation characteristics directly to metamodels (Heym 1993, Heym and Österle 1991, Harmsen 1997). An example of the former is the work of Punter and Lemmen (1996), who aim to apply a contingency checklist to define project strategies and method objectives. An example of the latter is Heym and Österle’s (1992) work where they collect characteristics of a method into a metamodel based on fixed classification schema. This classification includes parts focusing on the method (e.g. project management, risk management, system development), application type (e.g. expert, office or real-time system), and life-cycle of ISD (e.g. analysis, maintenance). The advantage of relating contingency factors to the metamodels is the close relation between factors and method components, whereas the external contingency lists are often not explicitly related to method knowledge.

The advantages of using contingency frameworks are obvious. They provide a high-level view of methods, and by covering several characteristics they help identify characteristics of methods which might not otherwise be noticed at all. Also, research has identified a wide variety of different ISD characteristics. However, the use of the contingency approach in great detail is costly as it requires a large amount of resources and skills to manage different methods and situational characteristics (Avison 1996, Kumar and Welke 1992). Its effective use is difficult, because it is almost impossible to classify, or to identify, relevant contingencies beforehand. Moreover, and general to all method development efforts, a combinations of various methods might be impossible because of different and possible conflicting underlying philosophies (Avison 1996).

To cope with the cost part, Punter and Lemmen (1996) aim to provide a ready-made contingency list. However, such a list operates at the general level of method knowledge, and does not allow other method choices than those already prescribed. Examples of the use of contingency frameworks deal with method knowledge in general, such as how much analytic modeling should be used instead of prototyping, or whether methods should be customized (e.g. Slooten and Hodes 1996). As a result, they can not be applied effectively in the detailed construction of methods. For example, it is unclear how factors like the stability of goals (in Slooten and Hodes 1996) or the amount of resistance can be related to the construction or even selection of modeling techniques. Similarly, most of the contingency-based ME criteria do not identify method knowledge in detail. As an example, the concept of inheritance is not addressed in any of the frameworks, although some of the mappings are relatively straightforward; such as single inheritance in class diagrams when a specific programming language is used. Because method knowledge is not addressed in detail, construction and maintenance of methods is not possible with the contingency frameworks discussed above.

Similarly, the cases related to detailed ME, such as customization of techniques or tasks of the ISD process, discussed in Section 2.5.4 (e.g. (Russo et al. 1996, Jaaksi 1997, Aalto 1993, Nissen et al. 1996, Kurki 1996, Tollow 1996, Cronholm and Goldkuhl 1994) or field studies about ME (Smolander et al. 1990) reveal that contingency frameworks for method selection are not used. Similarly, the reported cases of ME (cf. Section 2.5.4) did not apply any contingency frameworks. This does not mean that contingencies do not describe characteristics of applicable methods. Instead it indicates how difficult they are to use in the detailed engineering of methods. Finally, and maybe most importantly, it should be noticed that none of the contingency criteria have been validated for the task they have been proposed for. Although some of them have been recognized to be relevant in past projects (e.g. Slooten and Hodes 1996), their usability in method construction has not been clarified. As a consequence, we do not know whether they are relevant for method construction, which deals with detailed method knowledge.

An alternative and more detailed approach is to seek applicable methods or their parts based on their description and problem solving capabilities. This approach is closely related to the method knowledge behind modeling techniques: how can relevant aspects of the object system be adequately described with modeling techniques, and how they allow us to find alternative solutions. This approach is more widely applied in practice (e.g. Jaaksi 1997, Tollow 1996) because it is simpler and does not require as much knowledge and resources to carry out as the contingency approach. As emphasized by Jaaksi (1997, p 76), ME efforts take place under financial pressure to solve practical problems. Accordingly, work done by other method developers is considered as a contribution to the method only when it can be expected to produce a significantly better solution (Jaaksi 1997). As March and Simon (1958) have recognized, organizations tend to find satisfactory rather than optimal solutions. The significance of a method solution is evaluated based on the problems at hand, not only by characterizing contingency factors and their changes.

The use of problem-driven ME criteria in practice does not mean that it is better. It has, however, some other advantages in addition to those mentioned. First, since it is problem-focused and derives method requirements from the current case, it is not so idealistic as the contingency-based approach. Second, it is more open to new concepts of methods, because it is not restricted to applying existing method-related situation characterizations. For example, in Tollow (1996) difficulties in developer and end-user communication led to the development of more readable and understandable notations. The resulting modeling techniques were not available in other methods and thus had new concepts and rules. Third, it is more open to the requirements of detailed method knowledge. For example, in Jaaksi (1997) the interest in the modality of dialogs of the user interface was added to the dialogue diagram. This modeling technique was constructed by using the notation of state diagrams with added new concepts, e.g. to describe which of the windows is the main window, which are modeless, and what kind of tasks the user performs with the windows.

Problem-driven ME also has shortcomings. It focuses only on problems identified at the moment and provides little generalization possibilities through frameworks. As with contingencies, the formulation of a problem-driven method construction framework is difficult because of their generality and loose connections to detailed method knowledge. For example, the framework of Essink (1988), used by Punter and Lemmen (1996) in their MEMA model, characterizes the problem domain according to four levels of abstraction (i.e. object system, conceptual IS, data system and implementation) and eight aspects (e.g. goals, dynamics, process structures). Methods are then allocated to the same MEMA model for selection and construction. This approach would, for example, locate ER diagrams and class diagrams under the same problem solving situation. Thus, no distinction can be made between these modeling techniques, or between different dialects of them. Instead, an inductive approach has been proposed (Jarke et al. 1994): a framework should be developed from expected problems and applied to method refinements. This aspect is analyzed in more detail in Chapter 5.

Kumar and Welke (1984, 1992) address the importance of stakeholders’ values or design ideals as requirements of methods. A major difference and strength of their ISD-PVQ technique is the emphasize on participation in ME: the users’ requirements are the most important aspect of ISs and therefore also of ISD environments. Thus, the applicability of a method is considered according to how well it supports the method stakeholders’ (developers, end-users, managers, etc.) values and expectations of ISD. Simply, the methods developed are more easily accepted if they satisfy the requirements of the method users.

ISD-PQV has also been applied in practice (Kumar and Welke 1984), revealing the domination of technical and economical aspects of ISD at the expense of other values. The need to change the focus of methods from technical aspects of ISs is also noted by several other researchers (e.g. Lyytinen 1986, Avison 1996). The traditional perspective has been to see an IS as a technical innovation and focus less on behavioral and social consequences (Lyytinen 1986). The implication for ME is twofold: on the one hand, methods should include participative and social components to compensate for the bias towards technical and economic issues. Thus, stakeholder-driven ME approaches could be used to move the focus of the ISD group towards other values and design ideals. On the other hand, the current values of stakeholders are also major reasons for the dominance of “hard” valued methods.

The steps of ME can be supported by CAME tools. This means that the deliverables, metamodels and ME criteria, can be stored in and retrieved from the CAME repository, and methods can be compared, versioned, and adapted into a CASE tool. The most studied tool functions have been capturing method knowledge (cf. Heym and Österle 1993, Harmsen et al. 1994a, Verhoef et al. 1991), and building generic CASE toolkits which can be customized for different methods (cf. Teichroew et al. 1980, Chen et al. 1989, Sorenson et al. 1988, Bergsten et al. 1989, Smolander et al. 1991, Rossi 1995, Grundy and Venable 1996, Kelly et al. 1996). The latter type of tools also interest us since we believe that the results of ME efforts should be applied in ISD as a situational method. Because of this focus, we view both the method and the supporting tool as an end-product of the ME process.

The ISD tools can be further divided into two broad categories based on how they model the object systems (Lyytinen 1987). These categories include data-oriented and process-oriented approaches. In ME a similar division can be also observed: there is a variety of metalanguages and CAME tools that model the methods and support the storage of IS models made according to method definitions (Sorenson et al. 1988, Smolander 1991, etc.). In the process camp - with fewer representatives than the data-oriented camp - research has focused on process representations and tools which support the enactment of defined processes (Hofstede, et al. 1993, Wijers 1991, Hidding et al. 1993, Pohl 1996). Since our focus is on the conceptual structure behind modeling techniques we focus on data-oriented CAME tools.

In practice CAME technology is quite new. This can be observed from CAME research which focuses mostly on tool building rather than investigating their usefulness or usability (Tolvanen et al. 1996). For example, to our knowledge only Marttiin et al. (1993) have analyzed metaCASE tools more systematically in terms of their capabilities for establishing method-tool companionship. The earliest pioneer SEM, (Teichroew et al. 1980) was introduced only at the beginning of the 1980’s. Most current CAME tools are outcomes of research projects, including MetaView (Sorenson et al. 1988), MetaEdit (Smolander et al. 1990a, MetaCase 1994), RAMATIC (Bergsten et al. 1989), Quickspec (Meta Systems 1989), MetaPlex (Chen 1988), IPSYS Toolbuilder (Alderson 1991), and MetaEdit+ (Kelly et al. 1996). During the last years commercial CAME tools, such as IPSYS Toolbuilder, MetaEdit+ (MetaCase 1996a, 1996b) and Paradigm+ have also begun to appear in the market. Commercial CAME tools are called metaCASE tools[13] and we apply this term because of its wider use. Marttiin et al. (1996) present a framework for comparing and evaluating CAME functionality. Isazadeh and Lamb (1997) and Kelly (1997) review and make partial comparisons of sets of CAME and metaCASE tools.

MetaCASE tools use a set of primitives, which allow them to describe a given method quickly and provide a set of mechanisms to implement tool support for the modeled method. To establish method-tool companionship (cf. Section 2.3.2) the method must be described using the metamodeling language the tool applies. Ideally the metamodeling language used in method construction is the same as that required by the tool, but often the tool-related metamodeling language limits the adaptation (Cronholm and Goldkuhl 1994), or other less formal metamodeling languages are used during the construction and design phase. The result of a ME process is a customized CASE tool which can assist ISD. The customized CASE tool is expected to produce, through its support for the situation-specific method, positive effects on the resulting IS or on the process of its development. In this sense metaCASE tools provide a new approach to establish symbiosis between methods and tools, and offer more degrees of freedom in method and tool selection (Tolvanen and Lyytinen 1993).

The CASE tools developed should not be viewed only as tools for making abstractions. They should also include other functionality which are affected by the method: checking, form conversion and review (cf. Section 2.3.2) are all design steps which need to be taken into account during adaptation. First, checking of the models is always dictated by the underlying metamodel. Because some rules of the method knowledge can not be guaranteed or even checked at modeling time, but only after models are made, the tool adaptation also includes the implementation of consistency checking reports. Second, form conversions between different IS models (Fraser et al. 1991) or to programming languages are driven by the underlying metamodel. In fact, requirements to change metamodels often occur because of the demands to generate certain programming code or analysis reports (cf. Section 5.1.2). Third, review of ISD deliverables with end-users is largely carried out via the IS representations (i.e. notations). This may require the use of less formal notations, or simplified versions of the modeling techniques applied by IS developers (e.g. Tollow 1996).

Finally, an additional advantage of using metamodel-based tools is that we can apply them to collect information on method use. In other words, using metamodels we can examine in a systematic and rigorous fashion how developers perceive the IS, in what notation the system is described, and how the models are checked. For example, in relation to experience gathering the metaCASE tools can be used to find which method knowledge is used or not used during ISD. This aspect is discussed in more detail in Chapter 5.

The ME approaches described are proposed for representing various aspects of method knowledge and for constructing this knowledge to meet different kinds of situational requirements. The survey reveals a mechanistic view of ME approaches: ME aims to develop methods by specifying and constructing them like machines, and little attention, at least in the published ME literature, is given to the introduction and use of methods. The approaches analyzed (cf. Tables 3-1 and 3-2) also have a narrow view of the ME process and examine method knowledge only at a coarse granularity. Regarding the types of method knowledge at the metalevel (i.e. knowledge behind the methods of ME), most research has only focused on metamodeling languages and conceptual structures, and little effort has been expended on other domains, such as what is the ME process in greater detail or what decision and criteria are relevant to ME. Other more specific limitations of the ME approaches are discussed below.

First and foremost, almost all approaches assume a priori construction of methods. Although some of the metamethods (e.g. Brinkkemper 1996, Punter and Lemmen 1996, Harmsen 1997) acknowledge the importance of experiences, they do not propose principles for identifying, collecting, and analyzing experience-based method knowledge. If learning from method use is ignored, methods can not be maintained or redefined based on experiences. This shows that the approaches assume either explicitly or implicitly that contingencies and problems are known beforehand and they are stable during the use of the method constructed. Any changes after method construction, e.g. during tool adaptation, method introduction, or method use, are not incorporated into the methods. Although some of the ME approaches acknowledge the changes, they do not include any steps or provide any mechanisms for refining methods. These approaches do not fit with our view of method knowledge as evolutionary (cf. Section 2.5.4): methods have evolved and changed in general and in organizations, and there is no reason to expect that they would not evolve in future. To our knowledge, only Jarke et al. (1994) focus on analyzing method use as a part of ME. They propose a traceability model to record process-related experiences. Yet their a posteriori ME approach does not consider the refinement of other types of method knowledge based on such feedback.

Second, most of the ME approaches are biased towards the selection of ready-made method components (or fragments). Their construction phases mostly consist of composing existing method knowledge, and method choices other than those already available are not considered. As a result, the ME approaches expect that someone has already proved a method or its component, and it is known to be applicable in specific ISD contingencies or problem solving situations. This is paradoxical, since there has been little research evaluating methods according to criteria used in method construction.

Third, the criteria (i.e. contingencies, problem characteristics, or values) used in the ME approaches are far too general to direct detailed method construction. Because of this, the proposed situation characterizations do not support detailed analysis, construction, and refinement of method knowledge. At best, the characterizations can be used to “prefer” a certain collection of techniques and methods. For example, the approaches do not distinguish techniques of object-oriented methods from techniques of structured methods. Although these general driving factors of ME are important in understanding and structuring method knowledge, the examples of local method development show that methods are developed at a far more detailed level. Similarly, the studies on individual designers’ understanding and use of methods indicate that method knowledge is different at the detailed level (Wijers 1991): for example, method knowledge is applied differently even at the level of single concepts of a modeling technique.

To summarize, none of the ME frameworks provide explicit principles for collecting and analyzing methods a posteriori and therefore do not explain how method refinements can be carried out. In this sense, they aim to deliver a method in terms of a constructed method, while little attention is paid to analyzing how the method is used and whether it has been successful.

[12] Although numerous process models have been proposed for software engineering (cf. Armense et al. 1993), some of which can be used in principle to specify method-related processes, we restrict our attention here to those explicitly developed for ME.

[13] The terms CASE shell (Bubenko 1988) and metasystem (Sorenson et al. 1988) have also been used. These terms usually refer only to functions which allow the implementation of CASE tool support for a selected method.