Roger Clarke's Web-Site

© Xamax Consultancy Pty Ltd,  1995-2024
Photo of Roger Clarke

Roger Clarke's 'Data Retention as Mass Surveillance'

Data Retention as Mass Surveillance:
The Need for an Evaluative Framework

PrePrint of 27 November 2014

Published in International Data Privacy Law 5, 2 (May 2015) 121-132, at

Roger Clarke **

© Xamax Consultancy Pty Ltd, 2014

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at


National security agencies have prompted governments in many countries to require all providers of telecommunications services to retain vast volumes of data about all traffic passing through them. This paper briefly reviews the background to the proposals and why they matter so much, with particular attention paid to proposals in Australia during 2014. It is argued that the evaluation of data retention proposals is severely hampered by the absence of an agreed framework within which arguments for highly intrusive proposals can be assessed. A framework is provided for evaluating such proposals.


1. Introduction

Many governments have been endeavouring to force providers of communications services to gather data about their customers' electronic traffic, and store that data for lengthy periods of time. The purpose of this is to enable law enforcement agencies to gain access to data when they want it. These initiatives are commonly referred to as 'data retention' measures.

The European Union (EU) provided credence to the demands of national security agencies by means of Directives 2002/58/EC and 2006/24/EC. Laws have been enacted in many EU countries and elsewhere. On 8 April 2014, the Court of Justice of the European Union annulled Directive 2006/24/EC, on the basis that it violates Articles 7 and 8 of the EU's Charter of Fundamental Rights (CJEU 2014). Some countries appear to be flouting the judgement, or passing legislation that nominally works around it.

Under some circumstances, data retention may assist in prosecutions, in criminal investigations, and perhaps even in crime prevention. On the other hand, data retention is a form of mass surveillance, and involves very substantial increases in the power of law enforcement agencies. It arguably represents a greater threat to democracy than it does to criminals.

This document commences by providing a brief and necessarily superficial presentation of the nature of the proposals and the issues arising from them, primarily from an Australian perspective. However, the primary purpose is not to contribute to the already-substantial literature on data retention. Rather, the paper argues that there is a need for, and a lack of, a body of principles whereby such seriously privacy-threatening proposals can be evaluated.

2. Data Retention

This section provides historical background, and then outlines the proposal current in Australia during 2014.

2.1 Data Retention Schemes Since 2001

Several threads can be traced that have resulted in the emergence of data retention schemes. Historically, telcos kept telephone call records for a period of time, in order to support their own invoicing needs. Since the early 1990s, the voice-specific Public Switched Telephone Network (PSTN) has been in rapid decline, as voice calls have migrated to mobile/cellular networks and more recently onto other carrier mechanisms that support the Internet. Moreover, subscription-based fees are cheaper to administer than call-based fees, and hence carriage providers and ISPs have been drifting away from the storage of call records.

The most critical driver of change, however, has been the dominance of national security extremism since the 2001 terrorist attacks in the USA, and the preparedness of parliaments in many countries to grant law enforcement agencies any request that they can somehow link to the idea of counter-terrorism. The Australian Parliament, for example, had about 30 statutes in place relating to counter-terrorism in 2001, but passed over 50 further Acts in the following decade, containing hundreds of provisions, and perhaps a dozen more since then (APL 2007, Williams 2011).

European law enforcement agencies worked from the early 2000s to achieve approval for data retention schemes. This resulted in the EU Data Retention Directives (2002/58/EC and 2006/24/EC), which provided justification for highly intrusive schemes in a variety of countries. For reviews, see Taylor (2006), Bignani (2007) and Jones (2008). The UK has been a leader in the field since at least 2000 (Bowden 2002), with upgrades in 2007 and 2009. For a comprehensive review, see Bowden (2012). Endeavours to further extend the 2000 Regulation of Investigatory Powers Act (RIPA) stalled in 2013.

In the opinion of the European Data Protection Supervisor, "The Data Retention Directive is without doubt the most privacy invasive instrument ever adopted by the EU in terms of scale and the number of people it affects" (EDPS 2011, pp. 2-3). The UN High Commissioner for Human Rights concluded that "... the collection and retention of communications data amounts to an interference with privacy whether or not those data are subsequently consulted or used [and] Mandatory third-party data retention ... appears neither necessary nor proportionate ..." (UNHCR 2013, pp. 7, 9).

Even more significantly, in April 2014, the European Court of Justice declared the Data Retention Directive invalid, because it "exceeded the limits imposed by compliance with the principle of proportionality" (CJEU 2014). This had the effect of invalidating the current laws of many of the European countries that have legislated to impose data retention. There are also a number of impediments in international law (St Vincent 2014).

The UK government nonetheless persisted with what was clearly an illegal program. Three months later, it rammed the Data Retention and Investigatory Powers (DRIP) Act through Parliament and the monarch, all within one day, 17 July 2014. It is far from clear that this legislation will survive the inevitable court challenges, but it provides a figleaf of apparent legality in the meantime. Other European governments may instead ignore the pleas of their national security agencies and face the facts: data retention in Europe 2006-14 has delivered very little of any value, particularly in relation to national security (Berg 2014). A summary of the position in 30 countries in late 2014 is in MSLODS (2014).

It became apparent in mid-2010 that Australian law enforcement agencies also wanted such capabilities (Grubb 2010). Subsequently, it was established that the push had commenced in March 2008 (Keane 2013). Indeed, concern had been expressed about an Australian initiative fully three years earlier (APF 2005). A yet earlier appearance of the notion was in July 2003, in a Draft Cybercrime Code of Practice that was developed by the Internet Industry Association (IIA), acting as a proxy for the Attorney-General's Department (Waters 2006).

It is abundantly clear that these initiatives are driven by national security agencies rather than by political parties. In Australia, for example, the same proposal has been re-run through each successive Attorney-General since 2010 - Labor's McClelland, Roxon and Dreyfus, and currently the Coalition's Brandis (Berg 2014).

The first few rounds culminated in the Attorney-General of the day putting the proposal on hold, and retiring to lick his or her wounds. In August 2014, however, the next round commenced in earnest, when the Attorney-General's Department provided a 'Confidential industry consultation paper ' to an unknown list of corporations (AGD 2014). Respect for the document's confidentiality was extremely low, with a copy becoming public within 24 hours of its despatch by the Department. Unusually, it appears that no attempt has been made by the Government to pursue the miscreant.

The Government tabled a Bill in the House on 30 October 2014. The Bill has drawn severe criticism from many quarters. Despite the tendency for 'national security' initiatives to be passed by Australian legislatures with the minimum of fuss, it remained unclear in late 2014 whether the current Bill would pass the Senate.

2.2 Retention of What?

A range of legal provisions exist whereby an organisation is required to retain personal data. For example, taxation laws in various jurisdictions require both business enterprises and individuals to retain data relating to their taxable income for a period of 3-7 years, a significant amount of which is personal data. Meanwhile, financial services and other organisations in many countries are subject to required retention periods for financial transaction data, typically of 7 years. Other obligations to retain personal data arise in the contexts of employment and licensing.

However, the 'data retention' that is in discussion here relates not to standing or file data, nor to financial transactions, but rather to electronic communications. Intrusions into human communications have always been treated very carefully by legislatures, because of the many sensitivities involved, not least of a political nature. This has applied in succession to the mail, the telegraph, the telephone and email (Schaffer 1978, Freeman 2012, Kierkegaard 2005). Since the popularisation of the Internet, a great many people conduct electronic communications very frequently, and hence a record of those communications would be extraordinarily intensive and revealing.

At various times, proposals have been presented as being about the retention of 'data', and at other times about the retention of 'metadata', as distinct from the retention of 'content'. The term 'metadata' originated in the library profession, to refer to various items of data that describe aspects of other data, such as its originator, its publisher, and the language in which it is expressed (Clarke 1997). Analyses have been published demonstrating how substantial a profile of a person can be generated from metadata alone, without access to communications content (e.g. Guardian 2013).

In Australia, the 'metadata' term has the appearance of being a 'red herring' introduced by the Attorney-General's Department in an endeavour to deflect attention from the facts of the matter, and to make a massively intrusive proposal appear less so. The debate descended into farce during the third quarter of 2014, with the Attorney-General and the Prime Minister making completely mystifying suggestions about what the term 'metadata' means, and which 'data' is intended to be subject to retention requirements (Keane 2014).

When applied to communications, 'metadata' is a loose concept that is subject to highly varying interpretations depending on the nature of the communication services involved and the person making the interpretation. Debates about 'metadata' are an irrelevant sideshow. The only meaningful basis for debate about data retention proposals is declaration of the particular data that the proposal encompasses.

2.3 The Current Proposal in Australia

The situation in Australia during the fourth quarter of 2014 provides a valuable case study of the problems inherent in data retention measures, and proposals for data retention measures, throughout the world.

The telephone industry has for many years stored data about each telephone call in a 'call charge record' (CCR) or 'call detail / data record' (CDR). This contains such data as the calling and receiving phone numbers, the call start- and end-times and its duration, and whether a connection was achieved. Because that data was in existence for business purposes, law enforcement agencies have used various powers in order to gain access to it. One way in which a sensible conversation could arise would be if a proposal was designed specifically to ensure that the items in telephone call data records continue to be available to law enforcement agencies, despite changes in technology. However, one of the few aspects of the latest proposals in Australia that is clear is that their scope is far greater than that.

The extent of the uncertainties was underlined by a set of 10 questions asked by the Internet Society of Australia on 6 August 2014 (ISOC-AU 2014):

  1. What is the definition of metadata to be retained?
  2. Which entities are required to retain metadata (Retention Entity)?
  3. Whose metadata is required to be maintained?
  4. What method of metadata retention must a Retention Entity employ?
  5. When must metadata retention commence?
  6. Who will pay the cost of metadata retention?
  7. What authorisation will be required to access metadata?
  8. How long must metadata be retained and how will it be disposed of?
  9. Who will bear the risks of metadata retention?
  10. What ongoing review and reporting of metadata retention will occur?

From the viewpoint of Internet service providers, that list may have been fairly comprehensive. From a consumer perspective, on the other hand, it omits a number of important questions, such as:

  1. What security obligations must be imposed by the organisations that store the metadata?
  2. What uses of the metadata are permitted?
  3. Under what circumstances are those uses of the metadata permitted?
  4. Which organisations are permitted to make those uses of the metadata?
  5. What mitigation measures will be in place in order to protect against collateral damage arising from any aspect of the handling of the metadata?
  6. What controls exist to ensure that the safeguards are not nominal but real? In particular what sanctions exist, and what enforcement mechanisms?

Some of these consumer concerns were underlined in the response of the European Data Protection Supervisor to the judgement of the European Court of Justice: "... the Court has ruled that the retention of communications data should have been duly specified and the EU legislator should also have ensured that such data can only be used in very specific contexts. The retention of communications data for the purposes of the combat of crime should always be precisely defined and clearly limited. ... Among other things, the concept of serious crimes should have been more precisely described in the Directive and at the very least, basic principles governing access to and the use of the retained data should have been set out" (EDPS 2014, emphases in original).

Shortly after the ISOC-AU asked its questions, a Confidential Discussion Document, undated but apparently of 26 August, was provided, presumably to carriers (cf. telcos) and major carriage service providers (cf. ISPs) (AGD 2014). But the document was not provided to, nor even made available to, the public, nor to any civil society organisation. On 30 August, ISOC-AU requested the Attorney-General to "extend the consultation process beyond the very small sub-set of stakeholders you have involved to date, and, in particular, include ISOC-AU in this process". The letter went unanswered.

The August 2014 Discussion Document did not use the word 'metadata', and instead was phrased in terms of 'data' and 'content'. However, it failed to declare definitions of those key terms, merely providing examples and a list. In particular, on unnumbered p. 1 (my emphases): "The Department has previously provided high-level examples of what can be considered to be data, as opposed to content, to the Parliamentary Joint Committee on Intelligence and Security and the Senate Legal and Constitutional Affairs References Committee. Those submissions provided that data includes:

Further, "A mandatory data retention scheme will apply to only a prescribed subset of 'telecommunications data'" (unnumbered p.2). AGD (2014) claims that this term is "negatively defined" in the Telecommunications (Interception and Access) Act as "information or documents about communications, but not the content or substance of those communications" (unnumbered p.1). The term is not defined anywhere in the Act, and it is not used in the text of the Act, and it appears solely in the heading of a single section, s.181. The claim about a 'negative definition' appears to be based on s.172, which - in the hugely complex manner beloved of drafters of national security legislation - (possibly) prohibits disclosure of "information that is the contents  or substance of a communication [and of] a document to the extent that the document contains the contents  or substance of a communication", where 'communication' is undefined under s.5 but "includes conversation and a message, and any part of a conversation or message".

The document then outlined "a set of telecommunications data" that it intends requiring be retained by "all entities that provide communications services available in Australia". This is the usual vagueness that the Department builds into all of its documents, and has the effect, whether intended or accidental, of wasting a great deal of time and energy among policy analysts. The Department should not leave it to the reader to guess the scope of those expressions. It could mean, for example:

The "set of telecommunications data" was said to be "broadly consistent with the categories of data set out in Article 5 of the former Directive 2006/24/EC". That EU Directive was ruled invalid 4 months prior to the publication of the Discussion Paper, and remains invalid. Like its Five Eyes colleagues, the Australian national security community appears to regard illegality as at most a temporary inconvenience (Key 2012, TAP 2013, Thomas 2014).

The (confidential) "set of telecommunications data" list comprised 16 items under 7 headings (unnumbered pp. 3-8). Many were vague and potentially very extensive. Some of the items in the list were meaningless, or subject to highly uncertain interpretation in an Internet context, e.g. "the time and date of the start and end of the communication ..." and "the time and date of the connection to and disconnection from the service". The connection notion is rooted in telephony and the PSTN, and became progressively inappropriate as packet-switched networks took over from connection-based networks. It is capable of some kind of interpretation in some contemporary contexts, but in many others it merely invites misunderstandings.

In addition, many carriers and carriage service providers would not have access to some of the items, such as "the MAC address of the network interface used to originate the communication", and "the physical ... location of the device or equipment used to send or receive a communication".

The requirements included that service-providers retain "the identifier(s) allocated to an account, service and/or device to which a communication is sent or attempted to be sent" and "the identifier(s) of the line, device and equipment connected to the service to which a communication is sent", but with a qualifying statement saying that the (current) proposal "does not apply to or require the retention of destination web address identifiers, such as destination IP addresses or URLs". These statements were in such fundamental conflict that they left entirely unclear what obligations are intended to exist, and would at law exist.

The Government tabled a Bill in the House on 30 October 2014, entitled the Telecommunications (Interception and Access) Amendment (Data Retention) Bill 2014 (TIAADR 2014). The Bill has drawn severe criticism from many quarters, variously for its vagueness, the lack of concrete justification, the massive costs it would impose on ISPs, and the very substantial compromise of freedoms that it entails (Guy 2014, Daly & Rintoul 2014, Arnold 2014). It was reviewed, and severely criticised, in a joint parliamentary committee report (JPCHR 2014, pp. 10-22).

Since 2001, Bills sponsored by the national security community have in almost all cases passed through Australian legislatures with bipartisan support and at most tokenistic amendments. Previous data retention proposals, on the other hand, have failed to gain Ministerial support. It is unclear whether the current Bill will pass the Senate, where the Government needs the support of either the Opposition or a considerable proportion of the cross-bench Senators (Stilgherrian 2014).

3. Surveillance

The over-reach by many countries beyond their constitutional limitations shows a disregard for the law. Added to that, the ineptness of the drafting, at least in Australia, evidences a lack of technological understanding. The most serious issue, however, is that data retention proposals embody substantial threats to vital democratic interests. This section briefly traverses the key aspects of the concerns, expressed in terms of the Australian context.

3.1 The Need for Appropriate Powers

The starting-point is that it is essential that law enforcement agencies have powers available to them, to enable them to investigate criminal acts and prosecute the people responsible for them, and to monitor activities that are preparatory to the commission of serious crimes, especially crimes of violence.

A great many powers already exist, many of them well-justified and appropriate. These include the capacity to acquire search warrants for existing data, and to declare an intention to an organisation in relation to particular data, thereby imposing a legal obligation not to destroy evidence pending the arrival of a warrant, e.g. at federal level under the Royal Commissions Act (Cth) and in the States, e.g. under the 'document destruction' provisions of the Crimes Act (Vic) and Evidence Act (Vic). From time to time, deficiencies in existing laws emerge - variously because of poor drafting and changes in circumstances - and particularly as a result of technological change that undermines the intent of existing laws. Where the powers are appropriate, deficiencies need to be addressed.

An example of an updated law is in the area of preservation of records. In particular, it has long been an offence to destroy material if it is known that it "may be required in evidence in a judicial proceeding" (as expressed in s. 39 of the Crimes Act (Cth)). This was extended around 2000, when the notion of 'preservation orders' emerged, which oblige a person to "preserve and maintain the integrity of [specified] computer data for a period of time as long as necessary, up to a maximum of ninety days, to enable the competent authorities to seek its disclosure" (CoE 2001, Arts. 16-17). In Australia, 'preservation notice' provisions were inserted into the Telecommunications (Interception and ACcess) Act (TIA) in 2013, as ss.107G-107W.

However, a great many more powers have been added by the scores of so-called 'counter-terrorism' statutes passed by panic-stricken legislatures since 2001. Major problems exist even with some longstanding powers, and especially with the many additional powers that are continually being sought by law enforcement agencies. For example, the 2013 preservation notice provisions referred to in the previous paragraph go far beyond what the CoE Cybercrime Convention called for, in ways that are highly abusive of the privacy of Australians (APF 2011a, 2011b).

3.2 Categories of Surveillance

When considering data retention proposals, several different categories of surveillance need to be distinguished. For a framework for surveillance analysis, see Clarke (2009c). The base level is Personal Surveillance, which is of one or more specified individuals. It is vital that personal surveillance be authorised by law, primarily on the basis of prior demonstration to a judicial officer that there are reasonable grounds to suspect that the person has committed an offence or intends to commit a serious offence. In addition, a number of mitigation measures and controls are essential, in order to curb abuse, and limit unjustified harm to the interests of the individual and others.

A second category is Location Surveillance. Arguments can be readily mounted for suitably designed and managed monitoring of areas that are focal points of major vulnerabilities, such as the control rooms of power stations, and the perimeter fences of facilities that contain explosives. Some evidence also exists of benefits arising from the surveillance of ATMs. However, there has been an epidemic of location surveillance schemes, including a large proportion of CCTV installations, which are unjustified, badly designed and poorly managed (Gill & Spriggs 2005, Williams 2007, APF 2009).

Mass Surveillance is a major step beyond personal and location surveillance. It involves the monitoring of groups of people, usually large groups. In general, the motivation is to identify individuals who belong to some particular category of people of interest to the surveillance organization. It may also, however, be used for its deterrent effects. Mass surveillance is not justifiable in any targeted sense, but only on the vague basis that more data is 'a good thing'. In effect, the proponents of mass surveillance apply the 'original sin' tenet ('you're all guilty - we're just not sure what of yet'). Until 2001, democratic countries decried such attitudes as being a defining characteristic of un-free nations such as East Germany under the Stasi.

The laws relating to destruction of evidence, and the later extensions to create preservation notices, facilitate personal surveillance. Data retention proposals, on the other hand, are unequivocally a weapon of mass surveillance. The extent to which data retention proposals leap beyond accepted limits to the conduct of mass surveillance is encapsulated by this quotation: "It is incompatible with human rights in a democracy to collect all communications or metadata all the time indiscriminately. The essence of the freedom conferred by the right to private life is that infringements must be justified and exceptional" (Bowden 2014). Many other expressions of public dismay about data retention as a form of mass surveillance are in JPCIS (2013, pp. 147-161).

Recent years have seen the implementation of a rash of mass surveillance schemes. For example, Automated Number Plate Recognition (ANPR) schemes, particularly that in the UK, but increasingly also those in the USA and Australia, go well beyond the detection and prosecution of speeding and unregistered vehicle offences (Clarke 2009a). The anonymous use of aircraft was precluded by finessing commercial protections against a secondary market in flight-tickets into a 'counter-terrorism' measure. Anonymous use of public transport and toll-roads has been undermined by the withdrawal of cash payments without substituting non-identified card payment schemes (APF 2008).

Proposals for the retention of data about electronic traffic go even further than those threatening impositions. The scheme is proposed to be universal rather than scattered or in any way targeted. It would provide access not only to records of individuals' locations and physical behaviour, but also to strong indicators of their associations, interests and attitudes. Such schemes are highly intrusive not merely into data privacy and behavioural privacy, but into the privacy of personal experience (Clarke 1997). This is much closer to the spectre of 'mind control' than State control apparatus has ever before been able to reach.

3.3 The Abuse of Surveillance Data

Serious concerns also arise in relation to the security of whatever data would be subject to data retention requirements (JPCIS 2013, pp. 167-173). Such data would be extraneous to the normal business functions of most service-providers. The ongoing high volume of data breach notifications attest to the fact that the data security arrangements among organisations generally are already very lax (PRC 2014, ITRC 2014). Yet the huge volumes of highly sensitive data that would arise from a data retention scheme represent a concentrated 'honey-pot'. A honeypot attracts attacks from many sources, and hence even more substantial investment in security features would be necessary.

These new collection, storage and disclosure activities, together with the increased security safeguards needed for the data, would be a government-imposed cost, which each carrier and carriage service provider would need to minimise. Moreover, the scale of the costs that would be imposed on carriers and carriage service providers is so great that a further problem arises. The probability is that some service-providers would endeavour to 'monetise' the data that they are forced to store. That would be highly intrusive in its own right, and would inevitably result in yet more use and disclosure of highly sensitive personal data.

Perhaps the greatest source of risk, however, is abuse by government agencies. The appropriate standard for ensuring that powers are not abused is the preclusion of access without judicial warrants. Yet that standard has been whittled down by a long series of measures, including statutory demand powers, Ministerial certificates of expediency, and non-judicial 'warrants' self-issued by agencies.

In Australia, a key authority for access to telecommunications data, including telephone call records, is the Telecommunications (Interception and Access) Act (TIA) Part 4. The domestic intelligence agency, the Australian Security Intelligence Organisation (ASIO) is granted very substantial powers under Division 3 (ss. 174-176), other enforcement agencies under Division 4 (ss. 177-180), and foreign law enforcement under Division 4A (ss.180A-180E). Another source of uncontrolled and excessive powers is s.313 of the Telecommunications Act (TA) (APF 2014a). It has recently been revealed that the section is routinely abused by a wide range of agencies (Crozier 2013, APF 2014b). Beyond that, organisations that hold data sometimes even comply with mere requests, often without evaluating the reasonableness of the request, and in many cases with no basis on which they could form an opinion in any case.

There is also the prospect of 'function creep', and re-purposing of data that has been acquired. In one case, a UK law enforcement agency retained and exploited mobile phone call records given to it in error, and this despite the data relating to lawyers and journalists, both of which professions are particularly sensitive about breaches of confidence Nichols (2014).

In short, data retention databases would be accessed by government agencies through mechanisms that are subject to seriously inadequate controls. Moreover, a primary source of attacks on data retention databases would be national security agencies of other countries. In Australia, it is only to be expected that some of these will include the other four members of the 'Five Eyes' agreement - the USA, UK, Canada and New Zealand (PI 2013). The national security agencies of each of the five countries cheerfully subvert the laws of all five countries, including Australia's, by imposing surveillance on one another's citizens and sending the 'laundered' data on to one another (PI 2013, MacAskill et al. 2013).

Data gathered and stored under data retention regimes is subject to abuse by multiple parties. In particular, government agencies would routinely utilise their inadequately-controlled privileges in order to gain as much access to the mass of data as they saw fit, not as was warranted by the circumstances.

4. Principles for Evaluating Proposals

Of the vast array of post-2001 provisions that Australian Parliaments have meekly acquiesced to, most embody grave inadequacies in relation to justification, proportionality, transparency, pre- and post-controls, and audit. This has prompted civil society organisations to issue policy statements on democratic controls over surveillance by the State (NAP 2014, OM 2014, APF 2014c, EFA 2014).

However, the quality of debate about data retention proposals is undermined by a key deficiency in the processes of democracy. There is no imperative whereby proposals that have significant negative impacts on human rights are subjected to disciplined assessment processes. Instead of a calm debate based on evidence, consideration of proposals is dominated by the politics of parties and the crass populism of superficial public opinion polls. The outcomes of parliamentary processes are seldom optimal, and are frequently at least inappropriate, and sometimes distinctly irrational.

For the public interest in such matters to be effectively represented, it is necessary for civil society to promulgate Principles for the evaluation of proposals, and hold government agencies, governments and parliaments to account against those Principles. There is already a small vocabulary that is used by commentators as though there were established meta-requirements. These meta-requirements need to be formalised, agreed and published. An expression of such 'Meta-Principles' was proposed in APF (2013), and is reproduced in Table 1.

Table 1: Meta-Principles for Privacy Protection

1. Evaluation

All proposals that have the potential to harm privacy must be subjected to prior evaluation against appropriate privacy principles.

2. Consultation

All evaluation processes must feature consultation processes with the affected public and their representative and advocacy organisations.

3. Transparency

Sufficient information must be disclosed in advance to enable meaningful and consultative evaluation processes to take place.

4. Justification

All privacy-intrusive aspects must be demonstrated to be necessary pre-conditions for the achievement of specific positive outcomes.

5. Proportionality

The benefits arising from all privacy-intrusive aspects must be demonstrated to be commensurate with their financial and other costs, and the risks that they give rise to.

6. Mitigation

Where privacy-intrusiveness cannot be avoided, mitigating measures must be conceived, implemented and sustained, in order to minimise the harm caused.

7. Controls

All privacy-intrusive aspects must be subject to controls, to ensure that practices reflect policies and procedures. Breaches must be subject to sanctions, and the sanctions must be applied.

8. Audit

All privacy-intrusive aspects and their associated justification, proportionality, transparency, mitigation measures and controls must be subject to review, periodically and when warranted.

Naturally, effective application of these Principles depends on them being articulated into definitions, criteria and procedures, accompanied by examples of appropriate trade-offs among competing interests. The following section contributes to that articulation process, by applying the bare Principles to the data retention proposals in Australia current during 2014.

5. Application of the Principles to Data Retention Proposals

This section demonstrates the degree to which the processes in relation to Australian data retention proposals in 2014 fall short of reasonable public expectations. The sections correspond to the Meta-Principles as expressed in Table 1 above.

(1) Evaluation

A range of mechanisms already exist that implement the Evaluation Principle in more or less effective ways. Examples include the British notions of Green Papers and White Papers, the public-interest-oriented Environmental Impact Assessment process, and the more recent corporation-oriented Regulatory Impact Assessment process. Human rights issues are addressed by Privacy Impact Assessments, and by broader Human Rights Impacts and Social Impact Assessment methods. All of these occur prior to the tabling of a Bill, and are far more effective than Parliamentary Committee hearings.

In Australia, each time that data retention proposals come forward, they are uttered as government policy. To the limited extent that the proposals are subjected to evaluation, it is only after the government has committed to them, and indeed only after they have been expressed in convoluted legislative form, and tabled as a Bill. Moreover, such evaluation processes as are conducted are limited to Parliamentary Committees. Committee Reports may include some substantive analysis, but, to the extent that they do, it is overwhelmed by political factors.

(2) Consultation

Many guidelines exist for the effective involvement of stakeholders, including civil society and affected segments of the public. Some of these guidelines are published by parliaments or government agencies (e.g. EC 2014). A current arena in which considerable attention is being paid to 'multi-stakeholder governance' is policies relating to the Internet.

In the case of data retention proposals in Australia, such consultation processes as exist:

The government has been called upon multiple times by NGOs including ISOC-AU, Electronic Frontiers Australia (EFA), the Australian Privacy Goundation (APF) and civil liberties organisations, to conduct a multi-stakeholder consultation process. Yet the Attorney-General and his Department not only fail to engage with NGOs, but also routinely ignore correspondence sent to them on the matter.

(3) Transparency

Evaluation and consultation are undermined to the extent that inadequate information is available. Lack of transparency undermines trust, legitimacy and effectiveness.

As demonstrated in earlier sections, information about Australian data retention proposals has been obscured, partial and extremely unclear. During the 3rd quarter of 2014, the sponsoring agency actively sought to preclude civil society from seeing the proposal exposed to service providers. The incoherence of both public statements and the Bill make clear that the drafters have very little understanding of either the Internet or of the organisations on which they intend to impose requirements, and no understanding of or regard for public concerns.

(4) Justification

Any proposal with serious negative consequences needs to be justified by means of a systemic argument that demonstrates that the claimed benefits will be actually achieved, supported by empirical evidence. Inadequate justification lay at the heart of the European court's annulment of the EC Directive on Data Retention (CJEU 2014).

Generally, very little evidence has been provided in support of the contention that the measures are needed, and certainly nothing that demonstrates that specific aspects of the measures are necessary pre-conditions for the achievement of specific positive outcomes. It needs to be demonstrated that data retention results in outcomes such as successful prosecutions and interdictions of preparatory activities for serious crimes, which would otherwise not have been feasible. Instead, submissions by proponents for data retention are almost always vague assertions, depending on the catch-all excuse of 'national security' to avoid providing real information and evidence. Meanwhile, evidence exists that suggests that data retention is not effective in achieving its claimed aims (AV 2011).

The proponents for data retention, in Australia as elsewhere, make the assumption that, the larger the haystack, the better the chance of finding needles. That assumption is patently illogical, because needle-searching becomes harder, not easier, as the search-area becomes larger and busier. It could, however, appear logical to people suffering from paranoia, because larger haystacks would create more suspicious cases, and could give rise to the belief that there are more needles. Indeed, the kinds of loose data analytic techniques that are being applied to 'big data' are designed for the purpose of generating interesting propositions, nomatter how limited the evidence in support of them (Wigan & Clarke 2013).

(5) Proportionality

Beyond providing justification, proponents need to show that the positive outcomes outweigh the negative impacts and implications. Again, the European court, in its annulment of the EC Directive on Data Retention, quite specifically recognised this Principle, and concluded that "by adopting the Data Retention Directive, the EU legislature has exceeded the limits imposed by compliance with the principle of proportionality" (CJEU 2014).

The proponents of the Australian proposals provide no evidence that enables judgement about whether each privacy-intrusive aspect of the proposed scheme is commensurate with its financial and other costs and the risks that it gives rise to: "the scheme must be sufficiently circumscribed to ensure that limitations on the right to privacy are proportionate (that is, are only as extensive as strictly necessary)" (JPCHR 2014, p.13). The Parliamentary Committee went on to identify multiple specific aspects of the Bill that fail the Proportionality Principle. Little evaluation of alternative approaches is provided. The vague claim that data retention is 'a necessary tool' is undermined by evidence from Europe that it delivers very little of value.

(6) Mitigation

Deeply embedded in the notion of Privacy Impact Assessment is the obligation of the proponents of an initiative that has harmful side-effects to implement means to ameliorate those harmful impacts (Clarke 2009b).

Australian data retention proposals are characterised by a substantial absence of mitigation measures. This is particularly problematical because of the propensity for harm to individuals, to groups, and to social cohesion. This arises because of the high risk of 'false positives'. Actions taken by national security and law enforcement agencies against individuals are commonly conducted in secrecy and without disclosing the basis of the accusations. Individuals are reduced to arguing their innocence, without even a clear idea of what their crime purports to be. It is implicit in the proposals that this collateral damage is intended to be worn by the victims. A further serious concern is that the negative impacts are inherently discriminatory, based on religion and ethnicity. This feeds bigotries and raises tensions. Yet the proposals are not balanced by effective measures to build bridges to the affected religious and ethnic communities.

(7) Controls

A scheme that has satisfied the first six Principles has been demonstrated to be justified and proportionate, and to embody safeguards. However, the undertakings in relation to the design of the scheme are meaningless without assurance that they will be delivered upon. The scheme must also include mechanisms whereby the undertakings are, and are seen to be, complied with, and are effective in achieving their underlying purposes.

In the case of Australian data retention proposals, it is unclear what controls are proposed, how those controls will be policed, what sanctions will be available, and how those sanctions will be applied. On the basis of measures previously 'justified' on the basis of 'counter-terrorism' or 'national security', there is a strong likelihood that the controls would be grossly inadequate.

(8) Audit

All systems decay. In order to overcome this natural tendency, systems need to be subjected to periodic review, in order to detect ineffectiveness, inefficiencies, unintended consequences and abuses.

In the case of Australian data retention proposals, it is unclear what audit arrangements are proposed. On the basis of existing, completely discredited arrangements applying to measures previously 'justified' on the basis of 'counter-terrorism' or 'national security', there is a strong likelihood that the audit arrangements will be seriously inadequate.

6. Conclusions

From a public policy perspective, data retention proposals in many countries are seriously problematic, both technically and at law. In Australia, they have been, and remain, a comprehensive shambles.

For many decades, it was tenable to regard Australia's national security agencies as 'a bunch of bumblers' who did only limited harm to anyone, including 'the enemy' (McQueen 1997). In many countries, on the other hand, that comfortable image was never applicable. It is no longer appropriate in Australia either, because those agencies now have powers that are unjustified, disproportionate and uncontrolled, and feature seriously inadequate mitigation measures.

One Joint Committee of the federal Parliament noted that "A mandatory data retention regime raises fundamental privacy issues, and is arguably a significant extension of the power of the state over the citizen. No such regime should be enacted unless those privacy and civil liberties concerns are sufficiently addressed" (PJCIS 2013, para. 5.208 on p. 190). Yet it left the door open for the Department to pretend that data retention had the Committee's support (Recommendations 42-43, pp. 192-193). After the Bill was tabled, another Joint Committee expressed serious reservations about its provisions (JPCHR 2014, pp. 10-22).

It is essential firstly that the inanities of such schemes, and the serious lack of expertise among its proponents, become much more widely known, and that the dangers to democracy inherent in the proposals do as well. Because of dereliction of duty by its parliamentarians, "Australia has failed to make the principles of [ICCPR] directly enforceable in Australian courts" (Evatt 2003), and hence Australians lack the means to achieve a ruling from the High Court that would be equivalent to the European Court's overturning of the EU Directive. The proposals must therefore be beaten down in the court of public opinion, and beaten down each time that each new and compliant Attorney-General brings them forward.

Unfortunately, a serious democratic deficit exists, in that there is no imperative that ensures that proposals are subjected to disciplined evaluation. As the previous section showed, the body of Principles outlined in Table 1 can be readily applied to the 2014 iteration of the data retention proposal in Australia. Moreover, it can be readily shown that the proposal flouts virtually all of the Principles. There is accordingly scope for civil society to establish, and articulate, a set of Principles that can make good the current democratic deficit.


AGD (2014) 'Telecommunications data retention--Statement of requirements' Confidential industry consultation paper, Attorney-General's Department, August 2014, mirrored in many places, including

APF (2005) 'Review of the Regulation of Access to Communications under the Telecommunications (Interception) Act 1979' Australian Privacy Foundation, May 2005, at

APF (2008) 'Policy Statement re Road Tolls' Australian Privacy Foundation, September 2008, at

APF (2009) 'Policy Statement re Visual Surveillance, incl. CCTV' Australian Privacy Foundation, January 2009, at

APF (2011a) 'Cybercrime Legislation Amendment Bill 2011' Submission to the Joint Select Committee on Cyber-Safety, Australian Privacy Foundation, July 2011, at

APF (2011b) 'The Cybercrime Bill is Excessive and Unbalanced' Australian Privacy Foundation, September 2011, at

APF (2013) 'Meta-Principles for Privacy Protection' Australian Privacy Foundation, March 2013, at

APF (2014a) 'Revision of Telecommunications (Interception and Access) Act 1979 ' Submission to the Senate Standing Committee on Constitutional and Legal Affairs, Australian Privacy Foundation, March 2014, at

APF (2014b) 's.313 of the Telecommunications Act' Submission to the House of Representatives Standing Committee on Infrastructure and Communications , Australian Privacy Foundation, March 2014, at

APF (2014c) 'APF Policy StatementDemocratic Control of Surveillance by the State' Australian Privacy Foundation, May 2014, at

APL (2007) 'Terrorism Law' Parliamentary Library, Parliament of Australia, 28 August 2007, at

APL (2012) 'Telecommunications data retention--an overview' Parliamentary Library, Parliament of Australia, October 2012, at;fileType=application%2Fpdf

Arnold B.B. (2014) 'Data retention flopped in Europe and should be rejected here' The Vonversation, 7 August 2014, at

AV (2011) 'Serious criminal offences, as defined in sect. 100a StPO, in Germany according to police crime statistics ' Arbeitskreis Vorratsdatenspeicherung, 19 February 2011, at

Berg C. (2014) 'Going against the grain on data retention' The Drum, 12 August 2014, at 12 Aug 2014,

Bignami C. (2007) 'Privacy and Law Enforcement in the European Union: The Data Retention Directive' Chicago Journal of International Law 8 (Spring 2007) 233-238, at

Bowden C. (2002) 'Closed Circuit Television for Inside Your Head: Blanket Traffic Data Retention and the Emergency Anti-Terrorism Legislation ' Duke L. & Tech. Rev. 47, 1 (March 2002) 1-7, at

Bowden C. (2012) 'Submission to the Joint Committee on the draft Communications Data Bill' August 2012, at

Bowden C. (2014) 'Privacy and Security Inquiry: Submission to the Intelligence And Security Committee of [the UK] Parliament', 7 February 2014, at

CJEU (2014) 'The Court of Justice declares the Data Retention Directive to be invalid' Court of Justice of the European Union, Press Release No 54/14, Luxembourg, 8 April 2014 , Judgment in Joined Cases C-293/12 and C-594/12, at

Clarke R. (1997) 'Beyond the Dublin Core: Rich Meta-Data and Convenience-of-Use Are Compatible After All' Xamax Consultancy Pty Ltd, July 1997, at

Clarke R. (1997b) 'Introduction to Dataveillance and Information Privacy, and Definitions of Terms' Xamax Consultancy Pty Ltd, August 1997, at

Clarke R. (2009a) 'The Covert Implementation of Mass Vehicle Surveillance in Australia' Proc. 4th Workshop on the Social Implications of National Security: Covert Policing, April 2009, ANU, Canberra, at

Clarke R. (2009b) 'Privacy Impact Assessment: Its Origins and Development' Computer Law & Security Review 25, 2 (April 2009) 123-135, PrePrint at

Clarke R. (2009c) 'A Framework for Surveillance Analysis' Xamax Consultancy Pty Ltd, August 2009, at

CoE (2001) 'Convention on Cybercrime' Council of Europe, 23 November 2001, at

Crozier R. (2013) 'Attorney-General agency implicated in web blocking scandal' itNews, 30 May 2013, at,attorney-general-agency-implicated-in-web-blocking-scandal.aspx

Daly A. & Rintoul S. (2014) 'Europe says no to data retention, so why is it an option in Australia?' The Conversation, 14 April 2014, at

EC (2014) 'Stakeholder Consultation Guidelines' European Commission, Draft, June 2014, at

EDPS (2011) 'Opinion of the European Data Protection Supervisor on the Evaluation report from the Commission to the Council and the European Parliament on the Data Retention Directive (Directive 2006/24/EC)' European Data Protection Supervisor, 30 May 2011, at

EDPS (2014) 'The CJEU rules that Data Retention Directive is invalid' Press Statement, European Data Protection Supervisor, 8 April 2014, at

EFA (2014) 'Citizens Not Suspects' Electronic Frontiers Australia, August 2014, at

Evatt E. (2003) 'Bill of Rights and International Standards' Australian Journal of Human Rights 9, 1 (2003) 5, at

Freeman E.H. (2012) 'The Telegraph and Personal Privacy: A Historical and Legal Perspective' EDPACS: The EDP Audit, Control, and Security Newsletter 46, 6 (2012) 9-20

Gill M. & Spriggs A. (2005) 'Assessing the impact of CCTV' Home Office Research Study 292, Home Office Research, Development and Statistics Directorate, U.K., February 2005, at

Grubb B. (2010) 'Govt wants ISPs to record browsing' ZDnet, 11 June 2010, at

Guardian (2013) 'A Guardian guide to your metadata' The Guardian, 13 June 2013, at

Guy G. (2014) 'You want my metadata, George Brandis? Get a warrant' The Guardian, 6 August 2014, at

ISOC-AU (2014) 'Ten questions about metadata retention' Internet Society of Australia, 6 August 2014, at

ITRC (2014) '2014 Data Breaches' Identity Theft Resource Center, November 2014, at

JPCHR (2014) 'Examination of legislation' The Parliamentary Joint Committee on Human Rights, November 2014, at

Jones R. (2008) 'UK data retention regulations' Computer Law & Security Review 24, 2 (2008) 147-150

Keane B. (2013) 'Revealed: Attorney-General's drive for data retention law' Crikey, 3 October 2013, at

Keane B. (2014) 'Brandis' disastrous data definition reflects a confused government' Crikey, 7 August 2014, at

Key J. (2012) 'Prime Minister requests inquiry' NZ Government, 24 September 2012, at

Kierkegaard S. (2005) 'Privacy in Electronic Communication: Watch Your E-Mail: Your Boss is Snooping!' Computer Law & Security Report 21, 3 (2005) 226-236

McQueen H. (1997) 'Suspect History' Wakefield Press, 1997

MSLODS (2014) 'Update on How `The West' is Backing Away From Data Retention' MSLODS, 18 November 2014, at

NAP (2014) 'International Principles on the Application of Human Rights to Communications Surveillance', July 2013, rev. May 2014, at

OM (2014) 'Ottawa Statement on Mass Surveillance in Canada',, May 2014, at

PI (2013) 'Eyes Wide Open' Privacy International, November 2013, at

MacAskill E., Ball J. & Murphy K. (2013) 'Revealed: Australian spy agency offered to share data about ordinary citizens' The Guardian, 2 December 2013, at

Nichols S. (2014) 'UK cops: Give us ONE journo's phone records. Vodafone: Take the WHOLE damn database!' The Register, 26 November 2014, at

PJCIS (2013) 'Report of the inquiry into potential reforms of Australia's national security legislation' Chapter 5: Data Retention, pp. 139-193, Parliamentary Joint Committee on Intelligence and Security, 24 June 2013, at

PRC (2014) 'Chronology of Data Breaches' Privacy Rights Clearinghouse, November 2014, at

St Vincent S. (2014) 'International Law and Secret Surveillance: Binding Restrictions Upon State Monitoring of Telephone and Internet Activity' Center for Democracy & Technology, Washington DC, 4 September 2014, at

Schaffer D. (1978) ''Mail Covers and the Fourth Amendment: United States v. Choate' Loy. L.A. L. Rev. 12, 201 (1978) 201-231, at

Stilgherrian (2014) 'Australia's data-retention plans look increasingly out of touch' ZDnet, 26 November 2014, at

TAP (2013) 'NSA privacy rules broken thousands of times, Edward Snowden leaks documents show' The Telegraph / AP, 16 August 2013, at

Taylor M. (2006) 'The EU Data Retention Directive' Computer Law & Security Review 22, 4 (2006) 309-312

Thomas A. (2014) 'Jemima Stratford QC's Advice' All Party Parliamentary Group on Drones, 29 January 2014, at

TIAADR (2014) 'Telecommunications (Interception and Access) Amendment (Data Retention) Bill 2014' Parliament of Australia, 30 October 2014, at

UNHCR (2013) 'The right to privacy in the digital age' Office of the United Nations High Commissioner for Human Rights, December 2013, at

Waters N. (2006) 'Government Surveillance in Australia' Pacific Privacy Consulting, 2006, at

Wigan M.R. & Clarke R. (2013) 'Big Data's Big Unintended Consequences' IEEE Computer 46, 6 (June 2013) 46 - 53, PrePrint at

Williams G. (2011) 'A Decade of Australian Anti-Terror Laws' Melbourne University Law Review 35, 3 (2011) 1136-1166, at

Williams R. (2007) 'Orwellian' CCTV in shires alarms senior police officer'' The Guardian, Monday 21 May 2007, at


The paper was initially developed as panellists's notes for an event in Canberra on 11 September 2014, organised by Electronic Frontiers Australia. It was further developed as a contribution to the Necessary and Proportionate Week of Action coordinated by Electronic Frontier Foundation (EFF) later that month.

The paper has benefited from prior work by many privacy policy analysts, internationally and in Australia. Of particular importance have been contributions by Nigel Waters of APF and ACCAN, Holly Raiche of ISOC-AU, ACCAN and APF, Carly Nyst of Privacy International, Jon Lawrence of EFA, Bernard Keane of Crikey, and Caspar Bowden of The comments of a reviewer and the Editor greatly assisted in the clarification of the argument.

Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., and a Visiting Professor in the Research School of Computer Science at the Australian National University.

He was Chair of the Australian Privacy Foundation (APF), 2006-14, and has been a member of the Advisory Panel of Privacy International since its inception in 2000. He is Secretary of the Internet Society of Australia (ISOC-AU), and was a Board member of Electronic Frontiers Australia (EFA) 2000-04.

xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 65 million in early 2021.

Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 7 September 2014 - Last Amended: 27 November 2014 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2022   -    Privacy Policy