In this section we shall first describe the research approach adopted and relate it to published research on method engineering. Second, we formulate the research questions, and finally we describe the research method.

In this thesis our topic is ME principles for local method development. Reasons for selecting this topic are twofold: First, new situations and challenges of ISD, such as client-server architectures, object-oriented approaches, or business process re-engineering, necessitate the formulation of new methodical approaches.
Page: 15
Accordingly, instead of selecting methods from the collection of available ones (e.g. by using contingency frameworks) organizations are facing needs to modify and even to develop local variants of ISD methods (cf. Seppänen et al. 1996). At the same time methods must be analyzed, constructed, adapted into tools and maintained in a different fashion when compared to other method development strategies.

Second, current approaches to method selection and development do not provide adequate support for learning and creation of methodical knowledge. Hence, in this study local method development is viewed as a knowledge creation process which can not be done in a “one-shot” manner. As cases of local method development (e.g. Turner et al. 1988, Aalto and Jaaksi 1994) reveal, in-house methods do not remain fixed over time, rather they have a history with various configurations: parts of the methods are modified, some parts are excluded, and new ones are included. Therefore, methods must be seen as one part of organizational knowledge, which evolves and needs to be collected, maintained and shared. Based on this we argue that an important factor in local method development is the capability of an organization or a project to learn about method use and deploy this knowledge for method refinements. Thus, our research approach is anchored on the one hand in beliefs underpinning method engineering (Brinkkemper 1990, Kumar and Welke 1992) that focus on developing situation-bound methods, and on the other hand in theories of organizational learning and knowledge creation (Schön 1983, Nonaka 1994).

Before we formulate our research questions, we will conduct a survey of related research. This allows us to position our research within the context of ME research during problem formulation. In their prominent article Kumar and Welke (1992) describe ME and suggest four domains that have to be addressed in ME:

1) modular method construction,

2) stakeholder value based method composition,

3) need for computer aided support, and

4) organizational support for ME.

In the following each research domain is discussed in more detail and related research is described[5].

1) Modular method construction. Several researchers (cf. Kumar and Welke 1992, Harmsen et al. 1994a, Heym 1993) suggest that ME can be carried out by using pre-defined and tested method modules. These modules - often called a component base (Kumar and Welke 1992), or method fragments (Harmsen et al. 1994b) - help specify knowledge about ISD methods in two ways. They either describe a method’s static part through its conceptual structure, or the dynamic features of a method, i.e. its procedural part. The first aspect is incorporated in meta-data models (Brinkkemper 1990) which describe the conceptual structure of modeling techniques together with their representations. The latter aspect is defined by meta-activity models (Brinkkemper 1990), or by process models (Marttiin 1994, Jarke et al. 1994). These models contain knowledge about the stages and tasks of a method.

Most research done in this domain has focused on developing metamodeling languages (cf. Welke 1988, Wijers 1991, Smolander 1992, Heym and Österle 1992, Rossi 1998, Marttiin 1994, Harmsen et al. 1994a). Principles for using pre-defined modules and utilizing metamodels for method analysis and refinement have been far less studied. Here research has focused on comparing and combining metamodels (e.g. Hong et al. 1993, Henderson-Sellers and Bulthuis 1996b) and developing metrics for metamodel-based method comparison (Rossi and Brinkkemper 1996). Moreover, advances in metamodeling languages have mostly taken place in meta-data modeling (cf. Welke 1988, Smolander 1992), though some process models (Verhoef et al. 1991, Marttiin 1994, Jarke et al. 1994) as well as integrated meta-data models and process models have been developed (Heym 1993, Marttiin et al. 1995, Harmsen et al. 1994a). Major differences among these approaches can be found in their modeling power and capability, degree of formality, and ways to represent method knowledge. Because ME is a relatively new research field, there is a lack of experience in applying metamodeling and modular method construction principles. A few cases studying ME practices have focused on relatively small methods and mostly on the adaptation of methods to modeling tools (cf. Tagg 1990, Tolvanen and Lyytinen 1993, Cronholm and Goldkuhl 1994). Also some laboratory based experiments on representing method knowledge have been carried out (e.g. Wijers 1991, Verhoef 1993). However, they focus on individual aspects (i.e. how a single developer understands and uses a method) rather than on the use of methods in the large and by many. Hence, most studies reported on method modeling can be found from method comparisons and analysis (cf. Song and Osterweil 1992, Hong et al. 1993). For these reasons, the essential question: “How can we represent, criticize, analyze and refine method knowledge adequately to support local method development in practice?” has largely remained unanswered.

2) Stakeholder value based method composition. Because ME can be regarded as a change process, it is relevant that constructed methods meet users’ requirements. Hence, ME requires methods and guidelines to identify stakeholders - such as designers, programmers, IS users and managers - and their requirements (Kumar and Welke 1984, 1992). This, in fact, is an essential factor in accepting constructed methods. It can be expected that method users will more easily learn the methods, accept them, and use them if the methods are based on their requirements, in contrast to the situation where introduced methods are purely based on requirements outside the organization. The involvement of method users has been emphasized in recent method development efforts (e.g. UML, Booch et al. 1997) in which method user’s requirements and comments are collected more extensively than ever before. Although the participation is important it has not been studied as extensively: identification of stakeholders, dealing with conflicting requirements, and responsibilities in decision making are less studied in the ME literature.

In this research domain few empirical studies have been carried out. Goldkuhl et al. (1992) studied five CASE tool adaptation projects and identified different roles and needs for the tool adaptation. In this study, however, the research focus was on technical issues dealing with customizable tools rather than on local method development. Similarly, other studies of ME (e.g. Tolvanen 1995) have focused on a limited number of stakeholders and a few contingency factors.

3) Need for computer aided support. Another research stream in ME has focused on developing tools for capturing method knowledge (cf. Heym 1993) as well as building metamodeling-based tools that can be customized (cf. Teichroew et al. 1980, Chen 1988, Sorenson et al. 1988, Bergsten et al. 1989, Smolander et al. 1991, Rossi 1995, Kelly et al. 1996). These tools, often called CASE shells (Bubenko 1988), metasystems (Sorenson et al. 1988), or metaCASE tools (Kelly 1994), offer facilities to tailor CASE tools with desired methods. Hence, as ISD methods are supported by CASE tools, similarly metamodeling languages are increasingly supported by metaCASE tools. This symmetry has naturally introduced a more general term CAME (Kumar and Welke 1992, Computer Aided Methodology Engineering) to highlight the role of computer-based tools in ME.

As in CASE research (cf. Wynekoop and Conger 1991) there is a bias in ME research towards building metaCASE and CAME environments rather than evaluating them. There are many articles that describe either principles and requirements for such environments (cf. Marttiin et al. 1995, Harmsen et al. 1994a, Goldkuhl and Cronholm 1993, Heym 1993), or represent how one particular system has been implemented and how it works (cf. Teichroew et al. 1980, Sorenson et al. 1988, Bergsten et al. 1989, Chen 1988, Smolander et al. 1991, Rossi 1995). There is, however, a paucity of research that describes the use of these tools in practice. Only two empirical studies addressing the capabilities of adaptable environments was found[6]: Goldkuhl et al. (1992) studied method adaptations carried out with four different tools and five methods. Marttiin et al. (1993) made laboratory experiments by adapting the same method to three different CASE shells. These studies reveal that CAME tool developers have concentrated so far on techniques that allow tool adaptation rather than on developing techniques and principles for utilizing tool based knowledge about methods for example in method selection, method composition, construction, and reuse. Yet, without proven ME principles, the development of advanced tool support for ME will be slowed down.

4) Organizational support for ME. The use of ISD methods always involves a supporting organizational structure and mechanisms that ensure method selection, development, training, use, and maintenance. The key research question here is: “How should ME be organized inside a company together with its ISD efforts?”. This research domain is hardly tackled in the ME literature although methods are actually developed, taught and used locally (Wijers and van Dort 1990, Aaen et al. 1992, Aalto 1993): because organizations develop their own versions of methods, these tasks are already being managed somehow. Few discussions available (cf. Bubenko 1988, Tagg 1990, Tolvanen and Lyytinen 1993, Tolvanen 1995, Nissen 1996) study the roles and tasks needed for method engineering. Research in this domain has so far focused mostly on proposing an organizational position of a method engineer. Studies of the other people involved or tasks and organizational structures and mechanisms needed to carry out ME in practice are missing.

The goal of this thesis is to improve the situational applicability of ISD methods that forms a part of a modeling environment. This objective is examined as a problem of method engineering. Our special interest is in incremental aspects of ME. Any organization that builds ISs not only delivers systems as an outcome, but also learns and creates knowledge about ISD methods. In fact, knowledge on ISD and ISD methods is one of the most valuable assets in ISD organizations: methods can be seen as a part of organizational knowledge, which evolves and needs to be collected and shared in an organization. Consequently, creation of new knowledge about ISD methods can be characterized as an incremental learning effort in contrast to selecting methods solely in a “one-shot” manner and using them as readily applicable standards.

According to the incremental approach, an important factor in local method development is the capability of an organization or a project to learn about method use, externalize the experiences into explicit knowledge, and utilize the experiences for method refinements and knowledge creation (cf. Schön 1983, Nonaka 1994). In incremental ME method knowledge is managed by using metamodels combined with method experiences and supported by CAME tools. Our primary interest is not in how efficiently an organization develops ISs, but in how it creates information and knowledge about the ISD and about the ISD methods it applies. Our research objective can also be seen as an aim to develop methodical guidelines for ME. Method engineering is driven by a method, i.e. a metamethod. In fact, Kumar and Welke (1992) define ME itself as a “method for designing and implementing ISD methods”.

The motivation for our problem formulation is based on two observations: first, many organizations tend to develop their own methods, and second, there is a lack of principles and guidelines to carry out local method development (Russo et al. 1995). Although there is a plethora of methods available for ISD, hardly any could be found for local method development and for method engineering. To develop principles for method engineering the following research problem is formulated:

This question is divided into two more specific questions:

1) How completely can meta-data models represent knowledge about ISD methods for modeling tools? This problem can be defined as a method modeling (i.e. metamodeling) problem. It deals with the modeling power of metamodeling languages and inspects semantic data models as a basis for metamodeling. The problem is examined by seeking metamodeling language constructs to specify detailed method knowledge. Thus, this research question deals with extending support for metamodeling. We use the term meta-data model to denote a description of static method knowledge, in contrast to the dynamics of methods which are captured with process models or meta-activity models (Tolvanen and Lyytinen 1993), or with other type of metamodeling languages (cf. Section 3.2.2). Strategies for meta-data modeling include modeling of a single technique (i.e. its conceptual structure and representations) and integration of techniques into a method. We concentrate on meta-data modeling because most customizable ISD tools focus on changing the static part of method support, and similarly most reported cases of tool adaptation deal with specifying static aspects of methods (e.g. Tagg 1990, Goldkuhl et al. 1992, Nissen et al. 1996).

This question is important since appropriate metamodeling constructs are needed to describe the methods being developed and adapted (Wijers 1991, Brinkkemper 1996). Research in this area (cf. survey on ME research, Section 1.5.2) has focused so far on modeling single techniques or a relatively small collection of techniques. Moreover, if we want to apply metamodeling as a vehicle for method construction (Kumar and Welke 1992) and tool adaptation (Tolvanen and Lyytinen 1993) this question is of great importance: a detailed metamodel is a pre-requisite for developing tool support for a method. In terms of the steps of local method development (cf. Figure 1-1), this research question deals with method construction and tool adaptation.

2) How can experience of method use together with metamodels be applied for method refinements? Because knowledge in general (Nonaka 1994, Schön 1983) and of method use in particular is created by individuals, the ability to build up and capture experience is important for local method development. Our subject here is experience of method stakeholders (such as designers, tool experts, method engineers) which can be used to improve in-house methods. The question deals thus with principles of method refinement. Method refinement is investigated through a process of organizational learning (Schön 1983) in which experience about methods is obtained during method use, and knowledge is created through a continuous dialog with the collected experience and assessment of method use (cf. Nonaka 1994).

Two factors motivate this research question. First, method modeling has not been studied from the viewpoint of incremental method development, i.e. how experiences can be used for method refinement. The traditional approach (cf. Brinkkemper 1996) has been to construct methods once in the beginning of each ISD project rather than to provide mechanisms to gather experiences and relate them to the available method specifications. To extend the ME process we propose mechanisms for evaluating and improving the situational applicability of methods applied in modeling tools. In terms of the steps of local method development (cf. Figure 1-1), this question deals with advancing or refining methods based on experience. Second, empirical studies on method modeling and construction have been laboratory experiments or small cases (cf. Wijers 1991, Verhoef 1993, Tolvanen and Lyytinen 1993). Because of bias in individual developers (e.g. Wijers 1991), we lack knowledge of how organizations or teams develop their own methods. In this thesis we demonstrate the viability of the proposed incremental approach in two cases of method engineering. Thus, method development and method refinements are studied here in a longitudinal rather than snapshot manner, and on a project level. One reason for this can be found from our survey of ME research (cf. Section 1.5.2) which reveals that we lack knowledge of how ME efforts can be organized.

To summarize, this thesis puts forward some principles for incremental ME. These principles aim to systematize local method development. Our special focus is on the evolutionary nature of method knowledge. We argue that an important factor for the success of ISD methods is how an organization or a project creates and maintains method knowledge. In incremental ME metamodels can be used for capturing method knowledge, analyzing methods used, and refining methods based on available experience of method use. By finding answers to these questions, we can analyze available ME approaches and extend the principles and methods of ME. By doing so we can improve the flexibility of ISD methods and overcome the problems faced in the dominant “one-shot” introduction and use of standardized methods. In terms of domains of ME research (see Section 1.5.2), and the problems formulated above the thesis focuses on the first research domain: construction of methods based on meta-data models. The problem addressed, however, is also related to other research domains of ME. In the domain of tool support CAME tools can implement the proposed metamodeling capabilities as well as support experience gathering. In the domains of organizational support and stakeholders’ roles the incremental principles suggest how experiences can be collected and analyzed in an organization. Finally, whereas most studies on ME have focused on developing metamodeling languages and tools our study deals with the process of actual method construction and development.

Selection of research methods is always dependent on the research setting and problem. At the same time, problem formulation can be done in favor of a particular research method. In this thesis we apply two kinds of research methods. The first research method, used to study the metamodeling related question, “how completely can meta-data models represent knowledge about ISD methods for modeling tools?”, is conceptual: we model 17 ISD methods and validate their meta-data models by implementing methods in computer-aided tools. These method specifications are then used to analyze method knowledge as part of modeling tools and to extend languages for method modeling. This type of inductive approach has rarely been applied to such an extent for analyzing and developing metamodeling languages for ME (Tolvanen et al. 1996). Thus, the selected research method complements other research approaches applied (cf. Tolvanen et al. 1996).

The second question, “how can experience of method use together with metamodels be applied for method refinements?”, is studied both conceptually and empirically. In the conceptual part we analyze the literature on ME and relate it to the mechanisms of knowledge creation and organizational learning. In the empirical part we follow an action research strategy (Rapoport 1970, Susman and Evered 1978) also applied in IS research (Wood-Harper 1985, Jönsson 1991, Checkland 1991). The need for empirical approach is obvious because ME is a relatively new research area, and thus has received little attention to theoretical and research methodical issues. Especially in (meta)methods and ME efforts we could find neither reported cases nor systematic studies which aim to develop metamethods[7]. This observation implies that ME needs to be studied in its natural setting, i.e. in real life organizations. In other words, we believe that it would be difficult and hard to develop principles for ME in a purely deductive way.

In the study of incremental ME we examine two cases in which methods were developed and adapted to local needs. Both of these cases cover all the steps of local method development (cf. Section 1.3). They allow us to build a rich understanding of method development, and demonstrate the feasibility of the incremental approach. The main benefits of applying an action research strategy is to gain in-depth and first-hand understanding of the processes that take place in an organization in a natural setting. In the studies we gather requirements related to methods and capture this information in meta-data models. The data about an organization’s method development effort is collected by interviewing method engineers and users, and by observing the ME process. Also, the modified CASE tools are used to analyze the methods as they are supported with tools. As an outcome of the data collection we obtain different versions of methods (in terms of metamodels and adapted tools) together with reasons for method refinements. On the data analysis side, the explicit relation of method specifications to their changes offers a mechanism to indicate and explain method evolution.

In principle, surveys, field studies and laboratory experiments are all appropriate in studying method development. For example, in studying local method development and use of in-house methods, Russo et al. (1995) used surveys and Smolander et al. (1990) carried out a field study. Moreover, Wijers (1991) performed three laboratory experiments to study method knowledge as understood and used by ISD professionals. However, these approaches focus on obtaining a snap-shot view of practice, or do not offer a possibility to analyze the richness and detail of ME. Most importantly, they do not capture changes in ISD methods as well as an action research method does. In our opinion, these are of great importance when examining incremental ME.

The use of action research does not come without cost: The study does not meet the standards of “positivist” research because the approach offers few possibilities for statistical generalization and the researcher can not exercise control over experimental conditions. However, several researchers have advocated an action research approach in systems development research (cf. Galliers and Land 1987, Galliers 1992, Wood-Harper 1985, Checkland 1991), because the nature of method development and use of methods emphasizes a close interaction between theory and practice.

This study has several limitations. One notable limitation is the definition of an ISD method. As the title of the thesis suggests our view of methods is limited in how methods relate to modeling tools, such as CASE. We are interested only in those parts of ISD methods that can be modeled, formalized, and supported in computer-aided environments. Therefore, implicit or hidden parts of methods, such as their value orientation, are excluded from the study.

The second limitation relates to the focus on meta-data models and use of semantic data models. The former means that our interest in ME is only in static aspects of the method, namely the conceptual structure behind modeling techniques. The latter means that metamodels are developed on the basis of semantic data models which by themselves have limitations in IS modeling and presumably also in method modeling. The semantic data models are selected as a basis for metamodeling because they provide support for incremental ME in which maintainability, modularity, ease of use and support for communication among stakeholders are important. Moreover, most large metamodeling efforts (Hong et al. 1993, Heym 1993, Henderson-Sellers and Bulthuis 1996a, 1996b, Hillegersberg 1997) apply semantic data models, and most repositories apply semantic data models in their schema (CASE Outlook 1989).

The third limitation relates to the research method. Despite the benefits of action research studies, such as its closeness to the real world and focus on detail and change, the results can not be statistically generalized. Rather, they allow us to suggest conjectures (Yin 1993) on an incremental basis. The studies can demonstrate that the suggested ME approach can be useful rather than justifying it to be universally beneficial. A thorough examination of incremental, evolution based ME necessitates a longer time scale and larger samples than applied here.

[5] A more detailed analysis of related research can be found from Tolvanen et al. (1996).

[6] Most articles related to use of metaCASE tools (e.g. Tagg 1990) describe only the current adaptation product but do not evaluate the adaptation process.

[7] Although the literature offers some metamethods which include aspects of ME in addition to method construction, they focus on an a priori view of ME (cf. Section 3.2) and have not been validated or even demonstrated in real-world ME efforts (Tolvanen et al. 1996).