WAITING FOR THE GHOST TRAIN: STRATEGIES FOR MANAGING ELECTRONIC PERSONAL RECORDS BEFORE IT IS TOO LATE

[This is a pre-publication version of a paper that was delivered by Adrian Cunningham at the Aug 23-29, 1999 Society of American Archivists Annual Meeting in Pittsburgh, PA. It was subsequently published, with minor editorial revisions, in Archival Issues: Journal of the Midwest Archives Conference, vol. 24, no. 1, 1999, pp. 55-64. It is published here with the kind permission of the author and the Midwest Archives Conference. At time of writing, Adrian Cunningham was President of the Australian Society of Archivists and a member of the staff of the National Archives of Australia.]

Some History

In the world of electronic records research seven years is probably a couple of generations. Over the past seven years the literature on electronic records has consumed countless gigabytes of memory, experienced more than a few new paradigms, and travelled up any number of dead end streets.

Six years ago I published an article on the management of personal records in electronic form in the Australian journal Archives and Manuscripts under the title ‘The Archival Management of Personal Records in Electronic Form: Some Suggestions’.[2] While other articles on electronic records published at that time have since been swamped by a tidal wave of new literature, my 1994 article has been joined by only one other article on the same topic in the intervening period. There it sits in splendid semi-isolation, regarded with suspicion by some and apparently ignored by almost everyone else!

My aims in 1993-94 were modest. More than anything I wanted to kick-start a discussion of a neglected topic. I wished to address and redress the lopsided nature of the electronic records discourse which was, and sadly still is, completely dominated by the corporate/organizational records perspective. This lopsided discourse is something I have over the years criticized as ‘corporate myopia’, a phrase denoting the failure to recognize that private individuals also create records.[3]

It was my intention in 1994 to present a number of suggestions in the hope that they would get people thinking and talking about the issues involved. My suggestions were presented as being more tentative than definitive. I simply wanted to get some discussion and debate started in the hope that more detailed and authoritative strategies and solutions might follow.

A subsidiary aim of my 1994 paper was to get personal records archivists and manuscript curators interested in electronic records issues. My feeling was that my personal records colleagues considered the whole electronic records issue to be too hard. They seemed to find it easier to keep working with traditional paper formats, seemingly in the hope that someone else might solve the problem for them. In Australia this approach to dealing with problems is called ‘putting them on the never-never’. It was my intention to take the issue out of never-never land and put it squarely on the agenda of personal records archivists.

I believed then and still believe now that the issue requires urgent attention. We cannot afford to postpone dealing with the challenge of electronic personal records until tomorrow. This is because important personal records are increasingly being created in electronic form only. Waiting for a solution rather than working towards a solution effectively consigns into limbo those vital electronic personal records that are created during our period of inactivity. Make no mistake, there is an electronic records time-bomb ticking away out there in the land of personal records, and it is up to us to start working out how we are going to defuse it before it blows us all away.

By my own yardsticks, my 1994 article was an abject failure. The past five years have effectively been wasted. While my paper generated a few ripples of interest amongst personal records archivists in Australia, there has not since been a single article or piece of research in the Australian literature on this particular topic that has been written by a practitioner in the field. I could not even convince my own employing institution, the National Library of Australia, to take my suggestions seriously or to produce a viable set of alternatives. Instead my Library colleagues passed over my suggestions as being too anti-Jenkinsonian for their liking.

My article did, however, generate some interest among electronic records specialists in Australia. This had the positive effect of encouraging them to recognize that personal records are also records and therefore worthy of some consideration. The best example of this is a wonderful article by Sue McKemmish called ‘Evidence of Me….’, which appeared in the May 1996 personal records theme issue of Archives and Manuscripts.[4] This article was an attempt to take the study of personal recordkeeping back to first principles, by exploring the nature of personal recordkeeping and the broad social mandates for its role in paying witness to individual lives, in contributing to society’s collective memory, and in constructing our cultural identity. While McKemmish did not address the issue of electronic records per se, she set out a research agenda that should be pursued if we are going to devise adequate holistic strategies for managing personal records in electronic form. Sue McKemmish, however, is an academic at Monash University with a background in government records – I only wish that I could have provoked a similar response from a personal records practitioner!

Oddly enough, my 1994 article received more attention from personal records archivists outside of Australia than it did from my compatriots. The most detailed consideration of the specific issues it raised has been Tom Hyry and Rachel Onuf’s paper published in Archival Issues in 1997.[5] I have been delighted by this development and have had some interesting and illuminating email discussions with the co-authors. Finally, after all this time, the discussion I hoped to generate in 1994 seems to be getting under way – albeit on the other side of the world!

What follows is a re-examination of the suggestions I presented in 1994. Do those suggestions still hold water given what has changed in the wider environment over the past six years? Do my suggestions now look ridiculously misguided? Has my own thinking moved on? What would I suggest today by way of dealing with the same challenges?

In six years an awful lot has happened in the broader electronic records landscape. In that time there has been a shift away from pure theory and assertion to the development, implementation, and testing of practical solutions. Among other things this progress has included the emergence of a very encouraging and absolutely essential dialog with the vendors of commercial software systems and applications. Admittedly, progress has been on a somewhat limited front. It has addressed the short- to medium-term imperatives of capturing accurate, authentic, reliable, meaningful, and accessible electronic records in corporate/organizational recordkeeping settings. While this is a good start (and one always has to start somewhere), the challenge of dealing with the imperatives of long-term preservation and access remains considerable. Nevertheless, useful and interesting work is being carried out in the area of migration strategies and the use of stable, standard storage formats.[6] Given these significant advances, do my 1994 suggestions still stand up to scrutiny?

To recap, the previous article made four main suggestions. These were:

1. I rejected the non-custodial or ‘distributed custody’ approach to the archival management of electronic records that was then being recommended by David Bearman and which has since become an important component of the electronic records strategies of government archives in Australia. Bearman has described custody as an ‘indefensible bastion’, arguing that the best place to manage the migration of electronic organizational records is the business environment in which they have been created and used.[7] While distributed custody may be fine for government records, the transitory nature of personal records creators means that in Australia personal records archivists have little choice but to confront the custodial challenge if they are to preserve and provide access to important personal records created in electronic form. In saying this, however, I am conscious that I can only really speak from an Australian perspective, for in Australia we do not have the phenomenon of enduring family archives that are more common in the upper echelons of some European societies. All the same, even in Europe I would anticipate that distributed custody of personal records could only ever be the exception rather than the rule.

2. I argued in favour of what I called ‘pre-custodial intervention’ by personal records archivists to ensure that personal electronic records were properly created, managed and documented in the first instance, thus improving our ability to preserve and provide access to those records over the long haul.

3. In the area of custodial strategies I recommended the migration or conversion of electronic records to standard formats and the provision of online networked access to electronic records for remote users.

4. I highlighted the need for improved training in information technology for personal records archivists and the employment by collecting archives of specialist IT staff to assist with the technical work that is involved in managing electronic records.

What was the reaction to these suggestions and how do they look six years later?

From the limited reaction I have received I get the impression that few if any people had problems with three of the four above suggestions. The second suggestion is the one that has proved to be contentious. I will give further consideration to this matter shortly, but I should say at this point that I still stand by all of the suggestions I made six years ago. There are others that I would now add to the original mix (more of these later), but there is nothing that I said in 1994 that I would now recant.

The suggestion that personal records archivists should seek to become actively involved in the records creation process has been the one major bone of contention with my 1994 article.[8] It is a suggestion that I do not retreat from. Indeed, if anything, it is a suggestion that I feel far more strongly about today than I did then. It is very much in harmony with the so-called ‘records continuum’ school of thought that has emerged (some would say re-emerged) in Australia over the past decade. In continuum thinking there is no useful distinction to be made between records management and archival science – it is all simply ‘recordkeeping’. Continuum thinking posits that the division of records into the separate categories of ‘current records’ and ‘historical records’ impedes the pursuit of a holistic and integrated recordkeeping mission. Put simply, in continuum thinking a record is a record is a record – it’s just that some records need to be kept for longer than other records, and some records may end up being used by a wider circle of users than other records.[9]

Much of the impetus for continuum thinking has come from the emergence of electronic records. Continuum and post-custodial thinkers argue that the effective long-term management of electronic records requires more than a minor tweaking of traditional practices. The imperatives of electronic records are such that a whole new set of strategies is required. The old ways of doing things simply will not work in the digital networked environment. According to continuum thinkers archivists cannot afford to be the passive recipients of records that are no longer required by their creators. The traditional post-hoc approach to recordkeeping, which has probably always been unsatisfactory, is patently inadequate in the electronic environment.

The emergence of electronic records has highlighted the fact that we can no longer take for granted (if we ever could) that records once created will remain reliable, comprehensible, authentic, accessible, and durable for as long as they are required to be used. If electronic records are to survive as reliable evidence of human activity they have to be created and captured into well-designed, well-documented recordkeeping systems. These recordkeeping systems not only have to capture reliable records, they will have to be migrated across successive software and hardware platforms lest they become the useless casualties of the rapid cycles of technological obsolescence that is one of the defining characteristics of the digital age.

The design of durable, good-quality recordkeeping systems is something that cannot be left to chance. We have already witnessed more than enough electronic recordkeeping disasters where valuable records have been rendered useless or unreadable because of a lack of foresight and an absence of professional recordkeeping expertise during the system design and creation phases of the records life span. To quote one of my Australian colleagues “durable evidence-rich records don’t grow on trees”. Good recordkeeping requires the involvement of recordkeeping professionals throughout the entire life of the records. Put simply, if we are to have any electronic records to put into our archives we cannot afford to be squeamish about getting involved in the processes of records creation and recordkeeping system design.

This proactive agenda is something that government and organizational archivists are naturally more comfortable with than personal records archivists. There are good reasons for this, and I certainly do not underestimate the difficulties associated with pre-custodial intervention in the processes of personal recordkeeping. Governments and organizations are used to regulating the recordkeeping behaviour of their employees. They are also usually in the habit of employing recordkeeping professionals who can help to determine and administer recordkeeping policies and procedures. The personal recordkeeping domain is a far less regulated one. Personal recordkeepers are not normally in the habit of employing professional expertise to help them keep their records.

So, is the separation of archivists from the records creation process one of the defining differences between personal records archivists and corporate archivists? My feeling is that, if the answer to this question is going to be yes, then personal recordkeeping archivists are heading inexorably towards antiquarian oblivion. In the greater scheme of things this in itself may not matter very much. Where it does matter is that, if it indeed happens, we will lose forever a vital component of our documentary heritage – that component created by private individuals in all walks of life and in all areas of human creative endeavour. If we are to be true to our professional mission, we have a duty to do our level best to ensure that essential evidence of private human activity is captured into reliable and durable recordkeeping systems.

Objections Overruled

As far as I can discern there are three main objections to personal records archivists becoming involved in the creation phase of personal recordkeeping. These are as follows:

1: It is not always possible to discern the ultimate historical significance of records while they are being created[10].

This is a point that I am happy to concede readily. But this concession in no way undermines the validity of my argument. Quite simply, it is nonsense to argue that, because a strategy will not work in all cases, it should not be used at all. Of course no one could have known before November 1963 that the personal records of Lee Harvey Oswald would be of such all-consuming interest after his death. But it is equally true that it was apparent from a young age that Albert Einstein was going to be a major figure of the 20^th Century and that every effort needed to be made to preserve the records of his work and achievements. We cannot know what will happen in the future, but there are things about the present that we do know will be of enduring interest to society in the future. We should not be derelict in our duty to the future by neglecting those people in the present who we know are significant. It is a cop-out to argue that, because there are people in the present whose significance will only be revealed in the future, we should ignore everyone, even the Albert Einsteins of the world, until they reach the end of their lives.

Shifting the archival appraisal/selection decision closer to the time of records creation is not necessarily a bad thing in any case. It is in line with classical (as opposed to Schellenbergian) appraisal theory that records should be appraised on the basis of their contemporary functional context and significance, not through a process of second-guessing the shifting whims and trends of historical research.

Moreover, my exhortation towards pre-custodial intervention does not preclude the application of traditional post-hoc strategies for those individuals whose significance does not become apparent until late in life or posthumously. The application of such post-hoc strategies is likely to be far less successful in preserving complete and comprehensible records of the individual concerned, but it may nevertheless capture something of value to posterity.

2: Pre-custodial intervention is far too labor-intensive.

Certainly the provision of one-on-one guidance and assistance in personal records creation is likely to be labor intensive and is only ever likely to be pursued in the most significant of cases. All the same, some front-end investment of time is likely to pay dividends in the long run. Just think of all the time that is currently spent on the detailed arrangement and description of poorly maintained personal records fonds and how much time might be saved if the records arrived in the repository in perfect and complete order and under full intellectual control. A bit of time invested by the archivist earlier in the process is likely to save a lot more time that would otherwise be needed later on in sorting out the undocumented mess that is acquired by the archives – a process that I have heard described quite accurately I think as ‘picking up after the kids’.

Admittedly, some records creators will be difficult if not downright impossible to work with. Still, this is no excuse for not trying. As with any worthwhile endeavor there will be some successes and some failures. The trick is to minimize the failures through the application of patience and professionalism.[11]

In any case, there are more types of pre-custodial intervention than the provision of one-on-one guidance. Pre-custodial intervention can encompass a range of activities. It might include the production of guidelines on recordkeeping targeted at particular groups of personal records creators such as scientists, creative writers, etc. It might also include working with software developers and vendors to encourage the incorporation of good recordkeeping functionality and self-documenting features in the desktop authoring applications favored by personal records creators. These kinds of pre-custodial interventions hold the promise of getting the good recordkeeping message to the greatest number of personal records creators for a relatively small investment of well-targeted professional effort.

3: The involvement of archivists in the processes of personal records creation will lead to self-conscious and unnatural recordkeeping practices.

This is the Jenkinsonian objection to which I alluded earlier. Sir Hilary Jenkinson had an enduring attachment to the notion of recordkeeping as being an entirely objective, unselfconscious and natural activity. He deplored any occurrence of recordkeeping for posterity, arguing that such practices did not generate authentic, truthful, and reliable records. While Jenkinson’s ideas should not be dismissed out of hand, his emphasis on objectivity and truthfulness sit somewhat uncomfortably in our post-modernist present. Personally, I am less inclined to emphasise the Jenkinsonian ‘records as objective truth’ mantra than Terry Cook’s ‘records are contingent and need to be understood in the full context of their creation’ argument.[12] Certainly, I have seen plenty of examples of self-conscious recordkeeping in my time.[13] While the self-consciousness of the recordkeeping behaviour of the records creator needs to be discerned and understood by anyone using such records, it does not make them non-records – nor does it make them any less valuable as evidence, it is simply a different kind of evidence.

It has always intrigued me that the Jenkinsonian objection to continuum-based recordkeeping has been so common in the personal records sphere, but is virtually unheard of in the organizational sphere. Why should proactive professional recordkeeping only pose a philosophical problem for personal records and not any other types of records? One possibility is that organizations need professional recordkeeping assistance more for their short-term accountability and efficiency requirements than for the sake of posterity. Conversely, a collecting archivist advising a creative writer is only likely to be interested in the imperatives of posterity. This is perhaps the one defining difference between the regulated recordkeeping domains of governments and organizations and the largely unregulated recordkeeping domains of private individuals. Even here, however, there are exceptions. Some categories of personal recordkeeping are more heavily regulated than others. These regulations can take the form of occupational conventions, professional codes of conduct and other recordkeeping warrants. Architects and scientists, for example are required to keep good records or else risk losing their professional standing. Members of such groups may be more inclined to welcome some professional recordkeeping assistance without bothering to worry too much about what is in it for the professional recordkeeper.

So, while the Jenkinsonian objection cannot be discounted, it is not in my view a reason for inaction. Rather than deny the partiality or self-consciousness of records, these things should be recognized as reality, and the records should be managed accordingly with adequate contextual metadata that supports full interpretation and analysis.

The Future – it’s in our hands

Those readers who have read my somewhat tentative and exploratory 1994 article may have detected a greater degree of stridency in my current message. While in part this may reflect a greater confidence in my message, it is mostly a reflection of my frustration at the lack of activity and discussion in the area of electronic personal records.

While the rest of the archival profession has been busy reinventing itself to meet the challenges of the digital age, I get the impression that the collecting archives and personal records sectors of our profession, at least in Australia, have been operating on a ‘business as usual’ basis. I am, however, not without some optimism that my North American colleagues are more prepared to open up discussion on these vital issues than many of my compatriots.

Indeed, our procrastination may, ironically, turn out to be in some ways advantageous. This is because we may now be in the position to adapt some of the more successful electronic records strategies that have been devised in the government/corporate records sectors.

In Australia government and corporate recordkeeping is being transformed as a result of the widespread adoption of the 1996 Australian Standard on Records Management (AS 4390).[14] This world’s first national best practice standard has also provided the basis for a new international standard on records management, which is currently being finalized by the International Standards Organisation as ISO 15489. While neither of these standards has been written with personal recordkeeping in mind, it is my belief that the conceptual framework and recordkeeping strategies presented in these documents can be of enormous use to personal records archivists.

Taking the lead from AS 4390, the National Archives of Australia has developed a comprehensive suite of manuals, guidelines, and standards which are designed to assist Australian government agencies to design and implement recordkeeping systems to ensure the creation, retention, and use of accurate, reliable, and authentic electronic records.[15] Operating within a records continuum-based conceptual framework, AS 4390 recommends a rigorous methodology based on an analysis of the functions and activities of an organization and identification of the recordkeeping requirements or warrants for each of those functions and activities. Recordkeeping systems then need to be designed and implemented in ways that ensure that these recordkeeping requirements are satisfied.

There is no reason why this methodology could not be adapted to the design and implementation of personal recordkeeping systems. Indeed, an extremely useful research project could involve researching the generic recordkeeping requirements of particular categories of personal records creators – creative writers, lawyers, architects, politicians, etc. This research could then inform the production of booklets based on the AS 4390 methodology that would provide guidance to these individuals on how to create and maintain adequate records to meet their recordkeeping requirements. I should stress here that one of the sets of recordkeeping requirements accommodated by the AS 4390 methodology is ‘community expectations’, which encompass the cultural and historical imperatives of archives. These recordkeeping requirements need to be researched for each function or activity performed. In addition, the recordkeeping requirements embodied in laws, regulations, professional codes of best practice, and so forth should be researched by a combination of literature review and personal interview, in the manner pioneered by Wendy Duff in her research into recordkeeping warrants.[16]

For too long personal records archivists have been too ready to leave to pure chance the creation and retention of personal records documenting significant events and activities. Collecting archives have been built on the random remainders of those recordkeeping systems that just happen to be halfway decent.

The advent of electronic records presents us with a golden opportunity to improve both our societal recordkeeping practices and our professional documentation outcomes. To do this properly we need to conduct more research into the dynamics of personal recordkeeping, the societal warrants for personal recordkeeping, and the functional requirements for evidence in personal recordkeeping.[17] A good start would be some pilot research projects with some individual records creators.

Failure to pursue a more active agenda will leave us patiently waiting at the railway station for the goods train of life to deliver the unreliable electronic leavings of our society’s movers and shakers. Years of passive and patient pacing of the platform will come to an end, I fear, when the whistle blows and the train pulls in to the station and we finally come to the realization that it is full of ghosts and that ghosts do not satisfy our researchers’ need for solid, reliable, and authentic evidence of the past.