Linda Hill, Michael Kaiser, Jochen Kaempfer, Evelyn Kurz & Kay Fielden, UNITEC, New Zealand
kfielden@unitec.ac.nz
 

Hill, L., Kaiser, M., Kaempfer, J., Kurz, E. & Fielden, K. (2006, July), Managing Information Technology: A New Zealand Literature Review. Bulletin of Applied Computing and Information Technology Vol. 4, Issue 1. ISSN 1176-4120. Retrieved from

ABSTRACT

In this literature review the state of research in the Managing Information Technology (MIT) domain in New Zealand is explored. Forty MIT research papers were categorised and analysed according to sub-domain area and research method utilised. Emphasis has been placed on three areas: formal publication criteria for research papers (type of paper and publication type), research methodology, and research context. Initially each paper was appraised individually and then codified to make a statistical evaluation possible. This made the search for significant correlations of the different variables possible. The results of these explorations are also discussed.

Keywords

Literature review, managing information technology, research methods

1. INTRODUCTION AND BACKGROUND

New Zealand is in the process of transforming itself from an economical monoculture based on agriculture to a high-tech society in which knowledge workers are all important (Drucker, 1957).  Key to the success of such a transformation is successful research and development. Information technology research undertaken in both academia and industry is crucial for the development of technologically advanced products and management practices.

For this literature review forty articles (Appendix 1) about the management of information technology written in the period 2001-2006 were considered.  Thirty five of these articles were peer-reviewed. 

1.1 Search Systems

Thirty seven of the articles were retrieved from academic libraries including electronic databases, paper journals and conference proceedings.

Electronic resources included (Figure 1).

• EBSCO Host electronic database from which research articles on management practices in knowledge management and e-commerce were located
• NACCQ website from which conference proceedings, online journal articles, and, in particular, the special interest group on Business and Information Systems Management were found
• The ACM website was also accessed but was of limited use as it was difficult to locate research conducted in New Zealand
• Google Scholar provided an initial entry point for searching as well as a platform to retrieve articles that were not freely accessible on other electronic databases
• Google.  Having accessed Google Scholar first, it was often easier to find research articles on Google, for free

Figure 1. Retrieval of resources

Further online resources that were used included: Science Direct which was limited in its usefulness as membership and payment was required to access journal articles; IEEE also only offers full text articles for paying users; and publisher sites such as Taylor & Francis, and Springerlink also offer articles for a fee.  These resources were useful, however, to locate an article and its author/s.  It was then possible to go back to one of the other electronic resources to locate a free copy.

The ACM and NACCQ websites proved to be the most useful sites to retrieve research articles.  Twenty five out of forty articles in this literature review were located on free databases and 13 of the articles included were found on paying sites but then located free of charge on the internet.

1.2 Search Techniques

To locate articles within the Managing Information Technology (MIT) domain key words including ‘manage’, ‘management’ or ‘managing’ were used.  Within these searches further keywords ‘information technology’ (IT), ‘information communication technology’ (ICT) or ‘information system’ (IS) provided an intersection of both IT and Management.

By using special IT databases like ACM, searching solely for management papers was sufficient since it was assumed that all papers would be focussed on Information Technology.

To ensure that the papers were written between 2001 and 2006, some search engines offered a specific input field for a timeframe (for instance: ACM and Google Scholar offered a “from” and “until” field). When scrolling through conference papers, it was often possible to show all papers for one conference. This implicitly decreases the search results to a specific year. Finally, stating the specific year in the search form (maybe even connected with an ‘or’ - (‘2001 or 2002 or…’) also increased the chance that the paper was written in the required timeframe.

Locating research conducted in New Zealand was the most challenging criteria to fulfil. Since no way was discovered to search for this explicitly, several implicit ways were found. One way was stating either a city like Auckland or New Zealand itself in the search form. Nevertheless, New Zealand (or only Zealand) did not generate the desired results since the origin of authors is not always stated in the publication. Unfortunately, none of the common literature databases supported the search for author residency or affiliation. The papers were then reviewed manually to ensure that one of the authors was in fact affiliated with a New Zealand organization.

Another way of searching was to specify ’.co.nz, .ac.nz or .nz’.  In many academic articles it appears to have become common practice to state the author’s email address.  Searching for the domain name increased the chance of locating research that was conducted at a New Zealand organisation. Another approach was to search specific conferences within New Zealand, especially those of the NACCQ, since most papers were from New Zealand researchers.

Another approach was to search for a specific New Zealand author in the domain of MIT.  Also searching the citations in a specific paper generated useful results.

No articles were selected from e-books.

1.3 Publication Types

To analyse the 40 articles selected they were further examined for type of publication, the authors’ affiliation and the type of research. These 40 articles represented a subset of research published by New Zealand authors both nationally and internationally.

There appears to be several traditional forms for publishing academic writing: these are: as books on research; as conference publications; and as journal articles. More recently many publications have also appeared online as either online journals or conference proceedings. E-libraries have become commonly available and many academic researchers display their research papers online either on their own personal web pages or through institutional research pages.

1.3.1 Journal articles

Academic journal articles are based on a publication form that is periodically issued such as journals or specialist magazines. All articles that were found in (online accessible) journals like the “International Journal of Bank Marketing” were classified as “journal articles”.

1.3.2 Conference papers

According to Prytherch (1995) conference papers can be differentiated into proceedings or transactions that are defined by the time they are presented: before or after the conference. For this literature review, however, all conference papers are classified as “conference papers”.

1.3.3 Online publications

Research institutions and universities within New Zealand commonly publish their research in one or more of the publication forms mentioned above. In addition they use other resources - for instance an academic’s homepage - to publish their own work. Since the search for articles has been conducted through online information retrieval, those academic documents were considered as well. These are classified as “online publications”.

In addition, online publications can be reviews of institutions or companies that are issued less frequently, perhaps only once a year. One example of such a resource classified as an online publication is the “Health Care and Informatics Review” online web page published by Enigma.

Because of the possibility of multiple publication types pertaining to one research output, for instance, online publications that were originally conference papers or had been published as periodical articles,  the publication category used is that pertaining to the original form of the publication. One example of such an article was Nodder, Mason, Ateljevic & Milne (2003a). This article was published in the homepage of the New Zealand Tourism Research Institute as well as being part of the conference proceedings for the International Conference in Helsinki, Finland, 2003, the original paper being the Finnish conference paper.

Twenty three of the forty articles chosen were conference papers.  Twelve academic journal articles were also selected. There were five online publications chosen that were neither published in an academic journal nor in conference proceedings. Half of the articles found were published in New Zealand and the other half were published internationally.  All research projects were based in New Zealand.  Seventeen of the publications from New Zealand were freely available - compared to only eleven of the publications from abroad. Most of the international publications were conference papers (11/23). Twenty articles were published within New Zealand and just two out of five online publications were international.

It must be remembered that these results are based on the particular search methodology presented in this paper. If the search method chosen was to find an ICT conference covering an appropriate topic, and a further search had started from there, the number of conference papers found would not be random. Thus, it cannot be taken as fact that more than the half of all research conducted within New Zealand is reported in conference proceedings.

2. PEER REVIEW

Peer review is “the process of obtaining impartial opinions from the research and academic community in order to ascertain whether papers submitted for publications in journals or at conferences are of a suitable standard. The opinions are sought by publishers and conference organizers, and are requested from those whose expertise and stature are similar to the author’s.” (Prytherch, 1995, p.489)

Peer review is a system to edit and revise an academic report before publishing. According to Jefferson, Wagner & Davidoff (2002), peer review is an established practice for most academic publishers. However, the reasons for peer review may differ. The reputation of conferences and journals is, according to Jefferson et al (2002), one reason to establish peer review within the academic disciplines. Even if peer review is aimed at improving or guaranteeing the quality of academic research reports, there seemed to be little evidence of this. Jefferson et al (2002) suggest that there is little research conducted about the actual influence of peer reviewing. Overall, peer-reviewing depends on the publisher and the preferred method of peer-reviewing. In this literature review no distinction has been made on the type of peer-reviewing process employed by academic publishers: just that the article was peer-reviewed.

Thirty five of the forty articles were peer reviewed. These articles found were assumed to be peer-reviewed because of claims made by publishers on their website. For those articles where the peer-review process was not stated explicitly, peer-review was inferred from the submission process outlined on the publisher’s web page. One article was identified as peer-reviewed after email communication with the author. The peer-review process could not be identified for five of the articles selected.

Some of the search systems used, such as EBSCO Host, allow a searcher to choose “peer reviewed” as a criterion. Using this capacity, it quickly became obvious that there was no major difference for the results provided. It would appear that some articles on EBSCO Host have been labelled incorrectly.

It can be concluded from the articles selected for this literature review, that research in New Zealand is predominantly peer-reviewed.  

3. COPYRIGHT

With the growing use of the internet as a publication platform and the growing possibility of publishing content which is freely accessible, the protection of content and executive rights becomes increasingly difficult. In an age of copying and pasting information that is accessible world wide, it is essential to identify research as the work of the author(s) to protect intellectual property.

Of the articles found for this literature review 29 did not have any statement of copyright within the article itself. Eleven articles published by the ACM or the Australian Computer Society had an explicit copyright statement.

4. AUTHORS

An initial assumption that New Zealand’s small population would lead to just a few authors dominating research in MIT proved to be false. Only two researchers were involved in multiple articles.

Figure 2.  Number of authors

Considering the number of authors that composed one article together, it was found that nineteen of the articles were written by a pair of authors. Seven articles were written by a team of three authors, and three papers by a team of four researchers. Individual authorship accounted for eleven articles.

From this particular sample of MIT research papers, it appears that New Zealand’s researchers prefer to research in teams rather than individually. From analysis into the the field of study and the attitude of authors to research in teams, it can be clearly seen that the attitude differs depending on the areas of research. Research that covers science and technical subjects is predominantly conducted by teams, unlike the areas of language and humanities as they tend to be researched individually (Carpenter, Wallis, & Smith, 2001, p. 5) . As Managing of Information Technology is a subject with a technical background, the high number of authors working in teams is therefore not unusual.

There were only five articles out of forty that were written by a team which included international authors. All of those five articles were published abroad and not in New Zealand.

4.1 Affiliation

It was easy to classify the organisations or institutions with which the authors were affiliated because they were either from universities, polytechnics or institutes of technology. All other institutions or organisations were classified under “Company”. The differentiation was based on the names of the organisations: UNITEC as a research resource belongs to institutes of technology whereas the Victoria University of Wellington is classified within universities.

As it can be seen in Figure 3, more than two thirds of the articles found were written by authors affiliated with universities. Only six of the articles were written from within technical institutions, four for Polytechnics and three were written by authors that do not affiliate with any educational or research institution. 

Figure 3. Author affiliation

If an article was written by authors from one or more universities it was counted as one example for an affiliation with a university. If an article was written by authors from a university and from a polytechnic the article was counted as an example for both categorisations. As a consequence, the figure shows the ratio of how many universities or polytechnics were involved within the research and did not give the real number of universities involved.  

The amount of research that was conducted within universities and the few articles written by authors affiliated with organisations outside academia leads to the consideration of potential reasons. For example it must be acknowledged that publishers might have reasons to prefer to publish research from universities they have an ongoing collaboration with, rather than from companies that want to publish on a one-off basis.

In addition, the libraries that were used as search systems must also be considered as a potential filter for research that was available for this report.

To sum up, Figure 3 gives an overview of the research that was conducted in New Zealand and was accessible through particular libraries. However, it should not be interpreted as an argument that general research within New Zealand was mainly conducted by authors affiliated with universities.

Eight articles were written by academics at Victoria University followed by AUT, Auckland and Waikato with five articles, Otago and Massey with four articles and Canterbury with one article. It might be interesting to compare the number or research articles written by academics from technical institutions compared to the number of articles originating form each of the universities (Figure 4).

Figure 4. Overview of the amount of university research

5. TYPES Of RESEARCH PAPERS

5.1 Discussion Papers

According to Stevensons (1997) something “under discussion” is a “topic which is being talked about but which has still to be decided”. Discussion can thereby be differentiated from research when the article gives the reader an overview of the particular topic from different point of views, but does not state any final conclusion. A discussion is much broader than research or a report. Those articles the main part of which is a literature review and those that do not define the relevance of the outcome are classified as discussions.

 5.2 Reports

Stevensons (1997, p.132) states in her dictionary handbook that a report is a “formal document that discusses a particular subject or states exactly what happened”. The report is a kind of technical description of research that was conducted, combined with a literature review. It formally states the process or the results of a research/development. According to Prytherch (1995, p.547) a report carries a number that identifies it to be one out of a series of reports. In this research, articles without report numbers were classified as reports when they described development that was conducted rather than the research itself. 

Of the articles found, 27 were considered to be research papers. Only ten articles were reports and three were discussion papers (Figure 5).

Figure 5. Types of research

6. RESEARCH METHODS ADOPTED

This section is about research methods and the particular instruments of research methods adopted in the articles reviewed for this paper.

First, research methods including qualitative, quantitative research and mixed methods are discussed (Figure 6). Papers classified as ‘literature review’ are also included.  Next findings on research methods adopted are presented, followed by a discussion on data collection analysis and results obtained are presented.

Figure 6. Research methods

6.1 Types of Research

Page and Meyer (2000) define scientific research as research that is “empirical, public, repeatable and generalisable to other situations.” 

It is empirical when findings are based on observations or experiments and these methods have to be conducted systematically. Findings need to be public for others to revise and, when required, disagree or agree with the research observation. The conducted research must be repeatable for others to follow similar processes in order to have approximately the same or equivalent results. It should be possible to apply the findings outside the tested environment and to make them general for the overall population of the considered research field.

Scientific research is, after Page & Meyer (2000, p.14), the same as experimental research. All other kinds of research fell under the category of “non-experimental research”.

6.2 Research Approaches

There are two main approaches for conducting research: qualitative and quantitative. The approach used by any researcher is defined by the nature of the research, the results and the outcomes that the research aims for and the background and history of the particular field of interest.

6.2.1 Qualitative research

In qualitative research, a topic of interest is investigated in depth using explorative studies.

In this constructive approach, an inductive form of logic is used. In qualitative research, variables and categories that help to better describe a phenomenon emerge from informants of the study (Creswell, 1994, p. 7) .

To accomplish their goal of developing a theory, researchers collect data, formulate their hypotheses based on it and then test it using their data. (Frankfort-Nachmias & Nachmias, 1997, p. 249)

Qualitative variables used vary in kind, not in amount (Marczyk, DeMatteo, & Festinger, 2005, p. 49) they are about the verification of patterns or theories that help explain a phenomenon. Furthermore, the outcome of a qualitative study is context-bound information (Creswell, 1994, p. 7) . A theory that is developed in qualitative research is directly relevant the context, it arises out of its particular setting.

Thus, qualitative research is often used to determine research questions or hypotheses for subsequent quantitative research. Results are gathered through interviews and observations, formal measurement procedures are not involved in qualitative research. (Marczyk, DeMatteo, & Festinger, 2005, p. 17)

6.2.2 Quantitative research

Quantitative research attempts to find answers to “Why” questions; statistical analysis and numeric tabulations are used to measure the variables that determine a particular research question. Its approach is to use a deductive logic “wherein theories and hypotheses are tested in a cause-and-effect order.” (Creswell, 1994, p. 7)

Whereas in qualitative methodologies the variables emerge inductive within the study, in a quantitative research, categories and variables are defined “a priori” by the researcher and stay fixed throughout the study (Creswell, 1994, p. 7) , quantitative variables vary in amount. (Marczyk, DeMatteo, & Festinger, 2005, p. 49)

Findings may be obtained using statistical surveys; key features of this kind of research are formal and systematic measurements and the use of statistics. The goal of quantitative data is to “either falsify, modify or provide support for existing theories”. Deductively derived hypotheses are tested using statistical methods using the data collected (Frankfort-Nachmias & Nachmias, 1997, p. 249) .

6.2.3 Mixed methods

In combined methods, multiple methods of data collection are used. Creswell (1994) distinguishes between two approaches of mixing methods. “Within methods” combines data collection strategies of the same research method within one study. Alternatively, the “between methods” approach mixes data collection procedures of quantitative and qualitative research methods.

6.2.4 Literature review

Literature review has also been considered a method of conducting research. This is, when results of the study have been drawn from a literature review only and “does not present new primary scholarship” (Lyons, 2005) .

This approach can be used when there are several pieces of literature already existing that discuss a certain phenomenon.  The literature review can then result in a comparison of the results, identifying gaps or it can combine results that lead to further conclusions. A review of previous studies can show new ways how to interpret these or resolve conflict of results than seemed contradictory (Lyons, 2005) . 

In Figure 6 it can be seen that 18 of 40 articles reviewed adopted qualitative research methods. An example of a qualitative paper is Shakir’s (2003) article “Current issues of ERP implementations in New Zealand” which reports on an exploratory study.

Quantitative research has been conducted by eight researchers of the considered population.  Lang & Colgate’s (2003) “Relationship quality, on-line banking and the information technology gap” is an example of a statistically significant survey on customer satisfaction with on-line banking.

Twelve articles reviewed used multiple data collection and analysis strategies. Examples of mixed methods found in the sample can all be grouped within the “between methods” approach. Further, the main method used by these researchers was what Creswell (1994) defines as the “two-phase design”, whereby a study is conducted in separated phases: a qualitative and a quantitative one. In most cases of this sample, a qualitative data collection procedure was used to identify categories and variables. Based on these findings, the authors conducted surveys to statistically test the hypotheses derived in the first phase. Ng and Hope (2004) present a good example of a two-phase design of a between and combined methods paper: “Information requirements for customer relationship management: a case study in a New Zealand bank”. In stage one, a three-part questionnaire was design to generate a list of important issues, eight answers returned at this stage. Out of the results gathered in phase one, a survey was designed that asked participants to rate identified qualities using a 7-point scale, the sample size of forty-three was used to draw conclusions.

In two of the papers, a literature review was used to answer the research question.  Toland & Yoong (2005) showed in their paper “Learning regions in New Zealand: The role of ICT” that a literature review can be used as a research method. Firstly, they gave term definitions and described the significance of their research topic. Next, the concept behind the discussed issue and elements that influence it, are explained; citing expert opinion. Alternative models are elaborated, followed by examples that illustrate the discussed phenomenon. The authors then discuss the quantification of the research topic, relate their findings to the case of New Zealand and state the potential that can be tapped using literature to support their argument.

7. DATA GATHERING INSTRUMENTS

In order to collect data, different methods and approaches can be undertaken. The strategies that have been identified in the sample material, are as follows:  interview, field study, survey, and secondary data analysis (Figure 7).

Figure 7. Data collection procedures

7.1 Interview

The purpose of conducting interviews is to seek the judgement of experienced stakeholders who are affected by the research question (Shakir, 2003) . In interviews, individuals are being asked to contribute to the research by sharing their knowledge.

A common way of collecting information or data in an interview is to ask open-ended questions. This prevents the answers from being too directed. The goal of an interview is to obtain information relevant to the research question to reconstruct a participant’s experience.

Therefore, interviews are a method mainly used in explorative research questions rather than in analytical ones (Velde, Jansen, & Anderson, 2004, p. 102) .

7.2 Survey

A survey as an inquiry strategy is used to study a population sample in order to be able to generalise findings to the whole population. The purpose is to collect data from the largest sample size possible to produce statistic that infer characteristics of the general population (Page & Meyer, 2000, p. 114) . The results of a survey are quantitative (numeric) descriptions of aspects of the study population.  Data is collected from a sample rather than from the whole population. People are being asked question and these answers contribute the data being analysed in the surveys (Fowler, 1993) . Through the application of sampling it is possible to produce statistics that are representative for the whole population (Hakim, 2000, p. 76) .

Survey questions are generally formulated from concerns of the researcher and they can be asked in several formats. Closed format items as used in rating scales, checkboxes or true or false questions are easy to analyse. Free response formats give a richer source of information to the researcher but the effort analysing them is greater. (Bonebright, Miner, Goldsmith, & Caudell, 2005)

7.3 Secondary Data Analysis

Secondary data analysis takes place when already existing primary scholarship serves as the data source for a new study. Findings in secondary data analysis are based on data collected others (Frankfort-Nachmias & Nachmias, 1997 p. 304) .

 However, researchers do not use results drawn from former studies but use the raw data only. Therefore, results and conclusions drawn in studies using secondary data are based on re-analysis of data collected by others; the results are, ideally, not influenced by the results of the former study.

The focus of secondary data analysis is on “a distinctive contribution” of the data rather than “to extend or re-asses the findings” of the main study. (Page & Meyer, 2000, p. 24)

Twelve researchers used mixed methods or multiple inquiry methods, therefore the absolute number of data collection methods is more than forty because some research articles reviewed utilised more than one data gathering instrument.

Nineteen articles gathered data using surveys. In five of the articles, interviews have been used to collect data. Two authors gathered data by secondary analysis and there was only one field study.

Statistical surveys are often conducted as a second step that follows  qualitative research. In the case of research in “Managing IT”, common variables from qualitative research can serve as a basis for general surveys that find results with a greater validity to the whole population of interest.

8. CLASSIFICATION OF CONTENT

Classification for this literature review are shown in the literature map in Figure 8. Three subgroups were identified: those with a commercial background, those dealing with the management of resources and finally research conducted about the management of information (see also Figure 9).

Figure 8. Map for the classification of content

Subgroups within the commerce section included: E-Commerce; Project Management; Electronic Resource Planning (ERP); and Customer Relationship Management (CRM). E-commerce papers included concentrated on macro-economic issues rather than technical details. Within the domain of Project Management the focus was on specific companies, often using case studies. Those dealing with ERP (and, as a subgroup, CRM) also focus on specific companies or systems.

Resource Management consists of three sub-subgroups: Outsourcing, Human Resource Management (HRM) and Contract Management. The first sub-subgroup, Outsourcing, describes both overall risks and chances of IT outsourcing as well as the state of IT outsourcing in New Zealand. HRM describes all aspects of Information Systems which directly deal with people. Due to the fact that lots of New Zealand research is done on the educational use of IT systems, some of the literature was also within the education sector. Finally, Contract Management asks how the productivity of end users is affected by IT systems - especially Contract Management Systems.

The subgroups Network Management, Knowledge Management (KM) and Data Management all belong to the subgroup Information Management. Network Management describes various aspects of networked computing systems. This includes research about New Zealand network characteristics, distributed computing as well as network mechanisms to decrease email spam. The KM sub-subgroup covers all aspects of the management of organised information. Although some of the papers reviewed have a special New Zealand focus, like the Management of Maori tribe members, most do not especially focus on New Zealand issues. Again, many of the articles reviewed were based on a case study, dealing with a solution to a specific problem. Finally, Data Management covers information systems which organise data rather than knowledge. Compared to KM, the content stored is not organised within a specific context but only as organised values. Within this sub-subgroup, literature was focused on metadata as well as data retrieval and storage enhancements.

In the literature retrieved E-Commerce, HRM, Knowledge Management and  Data Management predominated. HRM appears to be one of the major research areas in New Zealand in MIT. E-Commerce, covering widely macro economical issues and the current state of E-Commerce in New Zealand, also seems to be a preferred area of New Zealand’s researchers.

Figure 9. Classification of content

8.1 Current State of Research

Most of the researchers in the sample of forty whose papers are related to “Managing information technology” conducted research that was about observing a phenomenon qualitatively rather than conducting cause-and-effect tests of a certain problem or issue. The non-experimental dominance in this case is explained by Page & Meyer (2000) “non-experimental research is by far the most frequently used type of research in management and business”.

In non-experimental studies, researchers observe existing circumstances to identify causal relationships, variables can neither be controlled nor manipulated nor measured in varying environments. Thus, a phenomenon is described with the relationships that exist between uncontrolled variables. (Page & Meyer, 2000) Both qualitative research and surveys are included in this category. There is a risk that qualitative research can “in extreme cases (…) be scorned as being meaningless in an environment where content and perceptions are what make the difference.” (Page & Meyer, 2000 p. 18) .

This argument is supported by Richardson & Hope (2003) who corroborate their choice of inquiry strategy (an interview) as follows:

“In the current study, interviews were found to be effective providing rich information that would be hard to elicit from surveys because knowledge is highly personal and is best negotiated face-to-face.”

Nevertheless, no study out of the sample of 40 papers used a scientific, experimental type of research. However, the type of research that was found in our study was mainly conducted for the purpose of solving a particular problem or to determine a certain phenomena.

Research that is conducted with “a specific application in mind” follows an applied research approach; this is what “academics and managers do to solve organizational problems”. (Page & Meyer, 2000, p. 19)

8.2 Research Quality

Even if the reason why the particular research is conducted is well explained, the aim of the research is often not clearly stated, in Delany (2004) , for example. Thus, the question of why particular research is conducted must be asked when there is no research aim mentioned and the conclusion does not provide any new findings. In many of the papers reviewed new ideas were mentioned but not investigated further and often the conclusion did not answer the research question. In the article by Enns, Huff, & Higgins (2003) different hypotheses were analysed but it was not explained why one of the hypotheses failed, even if overall the research was well reported.  

There is a high number of articles that were classified as “research” rather than “discussion” even when they do not conclude in an appropriate way. New Zealand researchers seem to hesitate to state the meaning of their findings and give straight advice.  

Regarding the large number of peer reviewed articles, it was assumed that the research articles have a common level of quality. When comparing the articles from Orr & Day (2004) with Kwok-wing & Pratt (2004) a wide gap in the quality of research can be identified. As Wagner. & Jefferson (2001) stated in their research on the use of peer review: “It does not always detect important work, nor does it reliably protect readers from fraudulent reports. “

In 37% of the studies considered for this literature review, a survey has been conducted and the results have been used to statistically determine the problem under investigation. Page & Meyer (2000, p. 114) consider surveys as the “most rigorous form” of non-experimental research. But, at least one condition has to be fulfilled: the sample size has to be appropriate and it must be sufficient in order to be able to generalise the results to the whole population of interest.

This means that the kind of research considered as best practice - being surveys- is definitely used to collect data for analysis. However, the findings would need to fulfill the one condition named above to be applicable, which is not the case in the articles named. Page and Meyer (2000) suggest that there is a danger of getting “extremely poor-research quality” in non-experimental research by not attempting to be scientific. As an example, the survey by McCarthy & Nesbit (2003) was conducted with only three participants.

8.3 Significance of Results

After Creswell (1994, p. 111) , the significance of a paper is given through (at least) three parameters: the value the study adds to scholarly research and literature in the field, the usefulness of the study to improve practice and the contribution of the study to improvement of policy.

The first question of how the paper or article contributes to scholarly research was often answered by authors in their papers by stating research gaps that have been identified. The attempt to fill these research gaps with their studies are clear rationales to contribute research that is of value to the scholarly research and literature.

For example, Viehland and Shakir (2005) state that “internationally there has been relatively little research that examines (research topic), and none in New Zealand”. Ng and Hope (2004) state that the “purpose of this paper is to fill this gap by surveying a group..” and Albertyn (2005) found that “limited research is available on (research topic).”

The improvement of the practice is dependent on the quality of the study and its results. Findings that provide a clear understanding of the problem investigated and that give answers on how to solve problems are of use in practice. But often research findings were directly related to specific environments so that an improvement of the practice was only possible for the particular environment.

Cash, Yoong & Huff (2004) give an example of a paper that clearly impacted and improved all three parameters. In this very comprehensive study, the changes in the business environment of IS professionals resulted in a lack of competencies which in turn results in barriers to managing IT. This lack is addressed to IS academics that need to teach new skills to IS professionals for them to be able to deal with the current IS environment.

8.4 Research Environment

Even though half of all articles were published overseas it is surprising that the number of international authors that join the research is so small. The reasons for that can only be assumed.

Another surprising fact is the low amount of research conducted by companies or non-educational organizations. Obviously, only big organizations can afford to have in-house development and research. Maybe this can be interpreted as being a result of New Zealand universities working closely together with industry. Thus, the universities do research in which industry is interested.

The following three criteria were used to judge the quality of the articles reviewed:

• proper conduct
• originality of work
• usefulness

Proper conduct includes the following of the appropriate codes of conduct, acknowledgement of intellectual property, the correct and unbiased gathering of data, the correct and unbiased use of statistics and other numeric methods (like simulations). It also includes the maintenance of high ethical standards.

Originality of work in this context means not only the acknowledgement of sources but also, how original the topic of research and its results are, i.e. how much they actually added to the body of knowledge (knowledge meaning information put to action) in their field. An example: there is only limited use in publishing a study conducted at an average high-school that has been conducted in a similar way on 500 different average high-schools shortly before, especially if the results do not differ significantly from the previous results and no other paradigms to support a novel interpretation are used.

Usefulness is rather difficult to judge and maybe the fuzziest of the above mentioned criteria.  Some research may not be useful per se or in its current state, but the underlying data gathered for it or certain aspects may be of use for other researchers. Obviously, the usefulness is tightly linked to the originality of a piece of research, but it might also be in such a remote or marginal field or so overly specialized that there will be no use for it in the foreseeable future. Usefulness therefore also includes the relevance: major research in the further development of the programming language design of COBOL might be original, but it is not useful, since COBOL is a language with rapidly fading importance and not in use for new projects anymore.

Most literature retrieved was classified as qualitative research, which was very often rather descriptive. This is due to the fact that the papers examined the current state of a specific area more than adding some “new” knowledge to this area. Apart from that, a large number of papers were based on a specific object, which was researched using a case study. Although those case studies were mostly done in great detail, the focus was rather small. In addition, opportunities to use this research for a wider scope are often missing. This would have been especially useful if the case study is not done to “explain” a wider topic but only to examine a specific system, for example the Membership Database (Charkova, Lin, Clear, & Lomax, 2003) .

Although the amount of pure quantitative research is rather small, it is often used to backup qualitative research - which makes it a mixed methods approach. A very good example for this would be Trotman ( 2005) . Very often the statistical analysis did not fulfil all of the aspects of an adequate statistical survey evaluation; deviations, significance and even the size of the probe and the general population were missing in more than one case.

The fact that a paper was peer reviewed did not change this fact. Overall, the quality of peer reviewed papers was no different from those which were not peer reviewed.

The highest quality appeared to be achieved in technical areas. Those papers were written in an adequate scientific style, backed up by quantitative research (which is documented adequately and fulfils statistical requirements). As well as this, those papers add scientific value to the researched area and can be used for a wider audience since they were rarely focused on a very specific or small research question, like the metadata research done by Cranefield ( 2001a; 2001b) .

Although all research was conducted in New Zealand, only about 30 percent actually focuses on New Zealand. Most of those used a New Zealand case study. Only a few focused on a general topic - like the network conditions for VoIP in New Zealand (Wen, Sathu, & Joyce, 2003) .

Although most literature has an explicit research question, some papers only state it implicitly. After reading the introduction, one normally knows what the paper tries to research, but nevertheless, an explicitly formulated research question would enhance the paper significantly. In addition, it would help the researchers to stay focussed and increases readability.

When it comes to timeliness, the papers differ heavily. Some papers are at the edge of research and help to push the edges further (Trotman, 2005) whereas other papers describe either existing systems or areas which have already been researched to a great extent (Watson, 2001) .

In contrast, some papers state a specific problem and the research focus, but rather than “solving” this problem they refer to papers in the same area and research already conducted. After analysing the existing knowledge in this field, the actual research is limited to only a few pages - summarizing the existing solution approaches (Dantin & Paynter, 2005) .

Overall, it can be stated that some research does not fulfil all scientific standards. qualitative research very often focuses on the actual research done in the area, adding only a small amount of new knowledge. Quantitative research, in contrast, sometimes is not conducted by statistical standards, but adds new knowledge to the research area. The focus of most literature is not New Zealand, nevertheless, most case studies are about New Zealand systems.

9. DISCUSSION

On reading and classifying the articles by form and content, it became quite obvious, that the quality and originality of the research varies strongly. While there were several papers that obviously contributed to their field, the value of other was rather uncertain or at least hard to determine.

It cannot be clearly stated that one specific research type dominates the research in the field of IT management.  There might be a certain tendency towards quantitative research, but the caveats concerning the selection of the articles, as given above, as well as the small sample size make the drawing of such a conclusion for the whole population very problematic.

Most articles in the sample were on a rather practical level. The quantitative research was usually descriptive in nature and did not attempt to devise a theoretical model to fit the results or allow any predictions. This, of course, makes it often an end in itself, since the lack of derivation of models from the data also means that the value of the research may be low for other researchers. All in all, the quantitative research seemed to be weaker that the qualitative research.

Some of the papers provoke mixed feelings, like that of Lang and Colgate (2003) : The study shows what common sense dictates: customers that are more or less satisfied with the way they interact with their bank are more or less inclined to change their bank, leaving other factors aside. The authors missed the opportunity to actually help the relationship management of banks since their conclusions appear to be trivial. Possibly their data could be used to foster other research efforts with a more detailed and focused approach. Even when sound methodology and originality is employed, the lack of usefulness makes a research effort questionable. There seems to be an inclination to create and conduct surveys and interviews, which might yield some interesting results but then to fall short on analysis of the raw data.

On the other hand, there were several articles that exemplified high quality research like Cockburn and McKenzie (2001) , which explored not only a technology, which can be considered cutting edge even five years after the publication, but also employed scientifically correct methods and drew interesting and surprising conclusions.

Some of the qualitative papers analysed were excellent, too. Especially those on library IT management by Bainbridge, Thompson and Witten (2003) and Buchanan and Hinze (2005) that show a sound knowledge of the specific field of research and leave no doubts that the respective authors have profound expertise. The papers have a well defined focus and there results are concise as well as relevant.

It is also noteworthy that the research community of New Zealand seems to be internationally well connected. A high proportion of research is done in association with authors from foreign institutions. Besides the advantage of English being the main language of New Zealand and the dominant language of the scientific community and the remaining connections to the other members of the Commonwealth, the multitude of connections to institutions abroad also underlines the quality of research done in New Zealand.

10. CONCLUSION

To sum up, the sample of forty papers of research related to “Managing Information Technology” by New Zealand authors has provided a wide range of research paradigms, methods and data collection approaches with a wide range of quality standards.

Due to the small sample size it would be equivocal to draw an absolute conclusion from the research done. It can only reflect the analysis of limited scope as conducted in this critique.

Nevertheless, it can be said that there are major research efforts in New Zealand and that there seems to be an acknowledgement of the importance of high-tech research in such a small country.

REFERENCES

Bonebright, T. L., Miner, N., E., Goldsmith, T. E., & Caudell, T. P. (2005). Data collection and analysis techniques for evaluating the perceptual qualities of auditory stimuli. ACM Transactions on Applied Perception, 2(4), 505-516.

Carpenter, J., Wallis, M., & Smith, N. (2001). Discovering research resources: researchers' use of libraries and other information sources. Culture Trends, 11(34/44), 1-34.

Creswell, J. W. (1994). Research design: qualitative & quantitative approaches. California: Sage.

Drucker, P. F. (1957). Landmarks of Tomorrow. New York, USA: Harper & Brothers Publishers

Engber, D. (2005). Quality Control.   Retrieved March 13, 2006, from http://www.slate.com/id/2116244

Fowler, F. J. (1993). Survey research methods. USA: Sage.

Frankfort-Nachmias, C., & Nachmias, D. (1997). Research methods in social sciences. New York: St Martin's Press.

Gay, L. R., & Diehl, P. L. (1992). Research methods for business and management. New York: Macmillan.

Gorman, G. E., & Calvert, P. J. (2000). Journal quality in the Asian region:  results of a pilot study for the IFLA round table of library and information science journals. In Conference Proceedings of  the 66th IFLA Council and general conference, Jerusalem, Israel. Retrieved March 3, 2006, from http://www.ifla.org/IV/ifla66/papers/156-125e.htm

Hakim, C. (2000). Research design - Successful design for social and economic research, 2nd edition. London: Routledge.

Jefferson, T., Wagner, E., & Davidoff, F. (2002). Measuring the quality of editorial peer review. Jama, 287(21), 2786-2790.

Lyons, K. (2005). Write a literature review.   Retrieved 12 March, 2006, from http://library.ucsc.edu/ref/howto/literaturereview.html

Lang, B., & Colgate, M. (2003). Relationship quality, on-line banking and the information technology gap. International Journal of Bank Marketing, 21(1), 29 - 37.

Marczyk, G. R., DeMatteo, D., & Festinger, D. (2005). Essentials of research design and methodology. Hoboken, NJ: John Wiley & Sons.

Page, C., & Meyer, D. (2000). Applied research design for business and management. Roseville, Australia: McGraw-Hill.

Prytherch, R. (1995). Harrod's Librarians' Glossary: 9,000 terms used in information management, library science, publishing, the book trades and archives management (8 ed.). England: Gower.

Sekaran, U. (2000). Research methods for business - a skill-building approach New York: John Wiley & Sons.

Stevensons, J. (1997). Dictionary of library and information management (1 ed.). Teddington, Great Britain: Peter Collin Publishing.

Velde, M. v. d., Jansen, P., & Anderson, N. (2004). Guide to management research methods. Malden, MA: Blackwell.

Wagner., E., & Jefferson, T. (2001). Shortcomings of peer review in biomedical journals. Learnder publishing, 14(4), 254-263.

Walsh, P. (2004). Research overview.   Retrieved 13.03, 2006, from http://www.vuw.ac.nz/home/research/overview.html

APPENDIX 1

Albertyn, F. (2005, July 10-13, 2005). Exploring e-commerce development: lessons learnt from a case study. In conference proceedings of the Eighteenth Annual Conference of the National Advisory Committee on Computing Qualifications (pp. 123-126). Tauranga: NACCQ.

Al-Qirim, N. A. Y., & Bathula, H. (2002, June 17-19, 2002). IT and eCommerce outsourcing in small to medium-size enterprises in New Zealand: an exploratory research. In conference proceedings of the Paper presented at the Fifteenth Bled Electronic Commerce Conference (pp.537-551).University of Maribor: Bled, Slovenia.

Bainbridge, D., Thompson, J., & Witten, I. H. (2003). Assembling and enriching digital library collections. In Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries (pp. 323 - 334). Washington, DC, USA: IEEE Computer Society.

Barnes, S., & Vidgen, R. (2003). Assessing the Quality of a Cross-National e-Government Web Site: A Case Study of the Forum on Strategic Management Knowledge Exchange. In Ralph Sprague (Ed.), Proceedings of the 36th Hawaii International Conference on Systems Sciences (Vol. HICSS 03). IEEE Computer Society

Buchanan, G., & Hinze, A. (2005). A generic alerting service for digital libraries. In Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries (pp. 131 - 140). New York, USA: ACM Press.

Cash, E., Yoong, P., & Huff, S. (2004). The impact of e-commerce on the role of IS professionals. ACM SIGMIS Database, 35(3), 50-63.

Charkova, R., Lin, A., Clear, T., & Lomax, T. (2003). Nga iwi o ngapuhi membership system: Relationship management and relational design. In S. Mann & A. Williamson (Eds.), Proceedings of the 16th annual NACCQ (pp. 31-40). Palmerston North, New Zealand.

 Chung, W., & Paynter, J. (2002). An evaluation of internet banking in New Zealand. In R. H. Sprague (Ed.), Proceedings of the 35th annual hawaii international conference on system sciences - track 7 (Vol. 7, pp. 2410-2419). Hawaii, USA

Clark, D., Bowden, S., & Corner, P. (2003). E-business in New Zealand 2002-2003: are we ready for the digital economy? University of Auckland Business Review, 5(2), 2-13.

Cockburn, A., & McKenzie, B. (2001). 3D or not 3D?: Evaluating the effect of the third dimension in a document management system. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 434 - 441). New York, USA: ACM Press.

Cranefield, S. (2001a). Networked knowledge representation and exchange using UML and RDF. Journal of digital information, 1 (8, Article No. 44, 2001-02-15). Retrieved March 5, 2006, from http://jodi.ecs.soton.ac.uk/Articles/v01/i08/Cranefield/

Cranefield, S. (2001b).UML and the semantic web. In Proceedings of the International Semantic Web Working Symposium ,Palo Alto, 2001, Retrieved March 12, 2006, from ww.semanticweb.org/SWWS/program/full/paper1.pdf

Dantin, U., & Paynter, J. (2005). Spam in email inboxes. In S. Mann & T.Clear (Eds.), Proceedings of the 18th annual conference of the national advisory committee on computing qualifications (pp. 33-38). Bay of Plenty, New Zealand.Delany, R. (2004). Smiling all the way.   Retrieved Feb 20, 2006, from http://hcro.enigma.co.nz/hcro/website/print_issue.cfm?issueid=52

Delany, R. (2004). Smiling all the way. Healthcare review online. Available: http://hcro.enigma.co.nz/hcro/website/print_issue.cfm?issueid=52 [2006, 20.02].

Dougherty, J., Clear, T., Cooper, S., Dececchi, T., Richards, B., & Wilusz, T. (2003). Information Technology Fluency in Practice. SIGCSE Bulletin, 35, 153 - 171.

Enns, H. G., Huff, S. L., & Higgins, C. A. (2003). CIO lateral influence behaviors: Gaining peers' commitment to strategic information systems. MIS quarterly, 27(1), 155-176.

Grant, P. (2005). The IT Industry Leads the Field in Knowledge Management: Yeah Right! Paper presented at the 18th Annual Conference of the National Advisory Committee on Computing Qualifications, Bay of Islands.

Henderson, S. (2005). Genre, task, topic and time: facets of personal digital document management. In Proceedings of the 6th ACM SIGCHI New Zealand chapter's international conference on Computer-human interaction: making CHI natural (pp. 75-82). Auckland, New Zealand: ACM Press.

Hine, J. H., & Dagger, P. (2004). Securing distributed computing against the hostile host. Paper presented at the 27th conference on Australasian computer science, Dunedin, New Zealand.

Hutchinson, M. K., & Weaver, C. K. (2004). Barriers to Women Studying Information Technology Courses. Bulletin of Applied Computing and Information Technology, 2(3).

Jesshope, C., & Zhang, Z. (2002). (EMAIL) A Content Management System for the TILE Managed Learning Environment. Proceedings of the Networked Learning Conference 2002, Sheffield, United Kingdom.

Kwok-wing, L., & Pratt, K. (2004). Information and communication technology (ICT) in secondary schools: The role of the computer coordinator. British Journal of Educational Technology, 35(4), 461-475.

Lang, B., & Colgate, M. (2003). Relationship quality, on-line banking and the information technology gap. International Journal of Bank Marketing, 21(1), 29 - 37.

McCarthy, C., & Nesbit, T. (2003). The impact of effective IT systems management on end-user productivity: an end-user perspective. Paper presented at the 16th NACCQ, Christchurch. incomplete

Ng, H. Y., & Hope, B. (2004). Information requirements for customer relationship management: a case study in a New Zealand bank. In conference proceedings of the 12th European Conference on Information Systems (retrieved April, 2006 from http://csrc.lse.ac.uk/asp/aspecis/20040118.pdf). Turku, Finland.

Nodder, C., Mason, D., Ateljevic, J., Milne, S. (2003a). ICT adoption and us in New Zealand’s small and medium tourism enterprises: a cross sectoral perspective. Information and Communication Technologies in Tourism - 2003. Frew, A, Hitz, M and O’Connor, P (Eds.) New York, Springer Wein, 355-364.

Nodder, C., Mason, D., Ateljevic, J., Milne, S. (2003b). ICT adoption and use in New Zealand's small and medium tourism enterprises: A cross sectoral perspective. Information and communication technologies in tourism.  [cited  25. 02. 2006]; Available from: http://www.tri.org.nz/NZTRI/Documents/ICTSME.pdf.

Orr, M., & Day, K. (2004). Knowledge and learning in "successful" IT projects: A case study.   Retrieved Feb 20, 2006, from http://hcro.enigma.co.nz/hcro/website/print_issue.cfm?issueid=52.

Pauleen, D. J., & Yoong, P. (2001). Relationship building and the use of ICT in boundary-crossing virtual teams: A facilitator's perspective. Journal of information technology, 16(4), 205-321.

Petrova, K., Philpott, A., Kaskenpalo, P., & Buchan, J. (2004). Embedding information security curricula in existing programmes. In Proceedings of the 1st annual conference on Information security curriculum development (pp. 20-29). Kennesaw, Georgia: ACM Press.

Potgieter, B. C. (2004 ). Change and innovation we expect of ICT teaching staff In Proceedings of the sixth conference on Australian computing education - Volume 30 (pp. 247-253 ). Dunedin, New Zealand Australian Computer Society, Inc.

Praagh, M. v. (2004). Information Technology, Dyslexia, and the Disabled Learner. Bulletin of Applied Computing and Information Technology, 2(1)

Savarimuthu, B. T. R., Purvis, M., & Fleurke, M. (2004). Monitoring and controlling of a multi-agent based workflow system. Paper presented at the Second workshop on Australasian information security, Data Mining and Web Intelligence, and Software Internationalisation - Volume 32, Dunedin, New Zealand.

Richardson, H., & Hope, B. (2003). The role of information systems and technology in case management:

             a case study in health and welfare insurance. The Australian Journal of Information Systems, 10(2).Retrieved Feb 25, 2006, from http://www.aaisnet.org/ajis/vol10no2.htm .

Shakir, M. (2003). Current issues of ERP implementations in New Zealand. Research Letters in the Information and Mathematical Sciences, 4, 151-172.

Taylor, H. (2005). The move to outsourced IT projects: key risks from the provider perspective. Paper presented at the 2005 ACM SIGMIS CPR conference on Computer personnel research, Atlanta, Georgia, USA.

Toland, J, & Yoong, P.  (2005) Learning regions in New Zealand: The role of ICT. International Journal of Education and Development using ICT [online],. 1(4). Retrieved Feb 25, 2006 from (http://ijedict.dec.uwi.edu/)

Trotman, A. (2005). Choosing document structure weights. Information Processing & Management, 41(2), 243-264.

Viehland, D., & Shakir, M. (2005). Making sense of enterprise systems implementation. University of Auckland Business Review, 7(2, pp. 27-36).

Watson, I. (2001). A decision support system for local government regulatory advice. In M. P. C. AAAI Press, US (Ed.), Proceedings of the 14th int. Flairs conference (pp. 329-333). Key West, Florida.

Wen, J., Sathu, H., & Joyce, D. (2003). Adaptive strategy for voice over internet protocol in New Zealand. In S. Mann & A. Williamson (Eds.), Proceedings of the 16th annual NACCQ (pp. 308-310). Palmerston North, New Zealand.

Yin, R. (1994). Case study research Design and methods Newbury Park: Sage.