±«Óãtv

I can't seem to get at that PDF, Nick - Firefox gives a page which looks identical to this one, Internet Explorer complains about needing to login to view it...

Why would you expect them to care about us? Public servants stopped thinking about service a long time ago. To them we're just people to be bossed around and taxed and generally harassed. And that attitude has been created by their political masters who, even now, are trying to justify their plans to spy on every aspect of our lives as part of their plan to control and know about everything we do. We've just got to say: No - Enough.

Argh! They just said that it breaches the Official Cover-Up Act to talk to journalist about this farce. (although I must stress I don't work for the government. I just heard a journalist say this on ITV News)

Does that mean I was wrong to tell Nick about B1 being the highest grade below SCS?

I was refering to the Bananas in Pyjamas. Honest.

There is another thing that worries me and that is that the password would be sent by email in clear text! If that was intercepted along the way, nobody would know but the files could be examined. However this is all academic as it is easy to break the passwords in a zip file - I've done it myself when I forgot!

pdf file not downloadable or readable at 18.56, 22/11

Like Nick I too am very concerned about the cavalier attitude to Data Protection legislation. Shouldn't the NOA as the guardians of good governance insisted that correct process and due diligence be pursued? Should they not of recognised that they were not allowed to see data for any purpose than for which it was originally collected?
Processes were flawed, processes were circumvented and the law flouted by the organisation who are supposed to report on the actions of government.
Who is left to protect us from inappropriate sharing of information between government departments. If the police want an extract of the HMRC database for a fraud case will they too get everyone’s tax records to accidentally forget to delete?

You're just muckraking and playing to the gallery, Nick. I can't see how these emails add anything to the management and data discussion. In fact, this dumb and chummy approach you're taking is retarding discussion and disappointing. I expect better.

I hope the government ignores this race to the bottom attitude and continues with its programme of delivery, and forcing responsibility down the chain so all stakeholders in Britain perform to a higher standard. Where do you stand on delivery? Think about that.

There is a chance the data may be safe despite what has happened.

There seems to be misunderstanding about password protection and encryption in many peoples's minds. In the sense of its use with data files a password is, in fact, a decryption key.

Now, if you have a password protected zip file and have forgotten the password then to only way to decrypt the files is using a guessing attack (sometimes called brute-force attack). Software tools exist for this.

However, the CD's in question will be merely coasters if non-dictionary password of 9 characters (alpha upper and lower case, numeric, special characters) or more was used.

A computer could guess about 20 million passwords a second. For 5 characters it would take less than a minute. For nine characters you are looking at about 10 years, for 10 characters over 500 years.

So, if the password was separately communicated, mixed all available keyborad characters and long enough then only the fastest computers would have a hope of cracking the code.

"Where do you stand on delivery?" In reception, waiting for a TNT van that never arrives?

The email dated 13 March mentions: "... not over burden the business by asking them to run additional data scans/filters that may incur a cost to the department".

Which 'business' would this be then? I thought that all parties concerned are public services acting on my behalf and paid for by us 'over burdened' taxpayers.

If this was a business the CEO's head would be on a platter by now.

Actually it does add something.

It confirms to the amusement (and dismay) of many that the files were almost certainly just zipped, as per the previous data set sent mentioned in the communication by NAO.

That means that unless they were still stuck using very old versions of whatever application they were using to zip the data (mostly likely Winzip knowing most cheap businesses, they don't go for lesser known applications, even if they're free OSS like 7zip, and I'd put money on them using Windows not *nix as it's an underfunded government office) then enabling 256 bit encryption would have been as simple as toggling a tick box in the dialogue, after all, they did state that the data was password protected, so that shows some conscious effort.

Breaking an encrypted zip file is a lot harder than breaking the old simple password protection on them, even if the same password was used.

Why has no one appeared to register that any half decent database is able to extract required data fields as easily as all data fields?

If HMRC's database is unable to do this what does that say about the millions of pounds spent on it? As my 14 year old says "more a Word document than a database ..."

Or is it just that HMRC's staff don't know how ... ?

I work in IT and I find the emails about filtering the data incurring extra costs amazing. Evidently HMRC do not employ anyone with the most basic database skills as this sounds like a trivial task. Even if they had had to call in a consultant for a day, what would it have cost? £1000 - nothing in comparison to what it's cost them now... especially with the completely pointless mailshot apologising to everyone (is HMRC entitled to free post? if not what's the cost of mailing everyone to apologise? I'd rather they didn't!)

Surely the crucial thing, Nick, as you have intimated, is that this email exchange neither explains anything nor supports the Govt's story. Nowhere in this exchange do any of the various parties say URGENT - THIS INFORMATION SHOULD NOT BE TRANSMITTED BY CDs IN MAIL. If the data had been cleaned by HMRC as requested, they would have sent to the NAO on disc millions of children's names with their associated National Insurance numbers. This would apparently have satisfied the NAO. If that disc had been lost, would it have been any less of a disaster than what actually happened? The whole tone of the emails indicates a systemic failure of awareness of the most elementary security by those concerned at both ends of the exchange, and makes the attempt to blame the 23 year old Assistant Officer who actually put the package into the post for this fiasco utterly preposterous.

You seem to be linking to a pdf containing the personal email and direct line phone and mobile numbers of a senior NAO official.

Seems a bit rum, when we are all excoriating the Government for their cavalier attitude to data protection.

No?

What the mails do show is that apart from sending the cd's out of the door there is no way any junior member of staff can be responsible.

Why do I say that?

Well the database extract was done by EDS (probably at vast expense if you note the comment that they have been able to do it without charging the NAO). There is no way that a junior member of staff could authorise them to do this work.

Nick, it's usual practise to link to a PDF file at the source, rather than copy it to your own server. Did you get permission to copy the file given the notices in the email footers? What will you do if the originator amends the original file with new information?

OK, so it can be argued that Joe Smith who however this official was didn't follow the rules, so perhaps it is not HMG's fault, but why did they not release the information as soon as they knew - why sit on it for a number of days during which a potential criminal could have been using it for nefarious purposes? No, Nick this has to remain in the headlines to find out exactly why the Government did not react to and make a statement sooner about the breach - irrespective of whether the official was "junior" or not!

There's a shadow in the background which you can glimpse here as through a glass, darkly - EDS and more generally the outsourcing of the HMRC's IT. It reads as if the HMRC are not in control of their own data, but have to ask (and hence pay) EDS to do anything that is out of the ordinary.

Of course, its incomprehensible that *anyone* in HMRC can even have unfiltered access to the entire database, let alone copy it onto CD, but one gets the impression that this was done out of frustration that they couldn't get the NAO the relatively simple dataset they asked for any other way.

Really, this supposedly difficult 'filtering' operation is the most basic database operation there is - it's simply leaving out certain columns in the query and limiting the number of rows. Any vaguely competent programmer with some basic knowledge of the file format (probably something basic like CSV) could have done this in an hour *from the data on the CDs* (which is what the criminals will be doing if they get their hands on it) - so if they had the whole dataset, why on earth didn't they do this basic filtering themselves?

The only explanation is that there simply isn't *any* deep IT capability left in the organisation and that it has all been outsourced. So the real issue this whole sorry affair drags up is not just that the civil service doesn't understand data security, but also that it doesn't really 'own' the data it collects from us.

This incident shows just how shambolic the civil service is and out of touch with the real world.
How is a junior 23 year old official able to download such sensitive information? How do we know he didn't make another copy?
What ever his bosses doing? What was the correct procedure? Why have we not been told what the correct procedure is? How many senior managers will be disciplined like they would in private industry?
Do the top civil servants even know what scrambling data is?
I do not blame the government as they cannot be expected to know everything that goes on to this level of detail but the civil servants do certainly need a kick up the back side.
Is it time for someone like Richard Branson to go in there and identify people who just aren't upto the job and need to go and go now.

I don't understand. Doesn't the distribution list of the email of 13th March 2007 15:23 demonstrate clearly that there were several people at HMRC - aka Benefits & Credits (some certainly senior from the high quality of the language used) who were aware that this data was being inappropriately sent.

That they took no action is an implicit confirmation that Senior Management at HMRC condoned and approved of what later transpired, and cannot be blamed on a Junior Official.

If you look at the attached email it refers to "a 100 zipped files" - shows just how educated the sender was. Muppet does not even come close.

I would not want to minimise the very serious cock up that occurred between these two offices, but I do think that (as usual) you read too much into the information that has been released. Anyone who has worked in a big organisation knows that the content of an email exchange is almost always only the tip of a big iceberg. A lot of organisational culture is just taken for granted in this sort of exchange. I am actually quite surprised that the NAO official actually went so far as to mention the need for care; he (she?) would have been very conscious of the possibility of upsetting his opposite number by even suggesting that he was not fully aware of the need for care. And in any event we don't know what had been said in the phone calls that parallel the email exchange.

BTW, I do object to your phrase "as spun in advance". In what sense was this factual statement an example of 'spin'?

2 things:

Firstly, the "ensure the CDs are delivered safely because of their content" sounds to me precisely like a concern for our privacy and data protection. Were you too busy laughing at these guys to notice that, Nick?

Secondly, it seems to me from these emails that the Govt story about the HMRC person being to blame is about right. The cost of encryption is nothing. The cost of recorded / registered delivery negligible. And to neglect all of this despite warnings about safety of data in transit seems astonishingly lax.

Of course, there's another option, which is that the culture is that appallingly lax throughout HMRC. But (unlike many a bandwagon moraliser in HM opposition) I don't feel I have an adequate basis on which to cast such a judgement.

And I don't feel the need to demand a scalp every time something bad happens (even something *really* bad, like this).

Where do I stand on delivery?

Where does TNT?

Don't see them getting any blame...

I regularly work with sensitive data, although in smaller volumes (a Local Education Authority's database of all the children in its area, including names and addresses of all children and some guardians) and on occasion, I have to send this data, sometimes in its entirety to an outside supplier. At least two of the LEAs I have worked with have such strict internal firewall policies, that it is impractical to upload data directly over the internet to the supplier. This leaves the option of burning it on a CD and posting it. The data is always encrypted, and always sent via a delivery system which is trackable. The recipient is warned to expect the CD and to let us know it has been received. We are always fully aware of the sensitivity of the data being transmitted, however, we are restricted in our tranmission methods by over zealous internal IT policies.

Interesting that the Revenue say that extracting the data would be too expensive. If the database is newer than 12 years old, I would expect it to be an Oracle Database. If that's the case, it would take me (a semi technical person that works in IT) about 15 minutes to extract the data once I had reviewed the data definition documents -- assuming they exist. Probably about 2 hours if the data definition documents were not accurate -- as if that's possible.

Finally, give it to a real Techie and you could, as a minimum, half those times, but of course with outsourcing you are talking about £15 per letter of a database query, working out at (probably) above £8,000 for the necessary work. Not that the person doing it would get anything, just an extra nugget for the suppliers bottom line

From NAO to NAO with ref to HMRC:

"Please could you ring ... so that he can pass on the password(s) in an e-mail"

Footer of HMRC e-mail:

"This e-mail may have been intercepted and its information altered"

Idiots, one and all. Also hilarious how they blank out bits of the e-mails, but forget to blank out the job titles of those involved, just to make it easier to hunt them down.

I wonder why the government of whatever persuasion doesn't employ IT professionals (I have always known that the NHS doesn't).

I'm impressed by the number of factual, grammatical and spelling errors in the emails and letters, e.g. the audit of "2004-04".

Your PDF of documentation released by HMRC includes signatures for Caroline Mawhood and Dave Hartnett!!! Seems odd to me.

Interesting that the Revenue say that extracting the data would be too expensive. If the database is newer than 12 years old, I would expect it to be an Oracle Database. If that's the case, it would take me (a semi technical person that works in IT) about 15 minutes to extract the data once I had reviewed the data definition documents -- assuming they exist. Probably about 2 hours if the data definition documents were not accurate -- as if that's possible.

Finally, give it to a real Techie and you could, as a minimum, half those times, but of course with outsourcing you are talking about £15 per letter of a database query, working out at (probably) above £8,000 for the necessary work. Not that the person doing it would get anything, just an extra nugget for the suppliers bottom line

So is it just me or has no-one picked up on the "last time we had 100 zipped files on two CDs".

So the sending of the Child Benefit database on a CD, outside of the HMRC building was not a one off, it had happened "last time" - presumably this time last year.

Stinks, doesn't it?

There are many issues that are concerning about this whole affair, like you mention the complete disregard for those whose civil servants allegedly serve. Not that this disregard is limited just to HMRC.

However as someone who works in the computer industry I find it unbelievable that they should be able to export 25 million records without top level authorisation. The question is who decided to give "junior" officials the ability to carry out such an operation on such sensitive data, surely a senior decision at some point you would hope.

Now we've advertised to criminals anywhere that if you get someone who's less than honest into that part of the HMRC then you've got a chance to get hold of data worth millions!

Not to mention the data protection act, bought in by this government, yet blatantly ignored by it's own officials.

I'd like to say it's unbelievable but given the utter disdain senior politicians appear to hold for the general public it's entirely believable, which is a very sad state of affairs.

Interesting that the pdf protects most email addresses, but has placed the Assistant Auditor General's landline, email address and mobile telephone number in the public domain.

So - pick a sample and tick it. Make sure you have the whole population to hand from which the sample has been taken so as to be happy the wool hasn't been pulled over your eyes.

The work of the auditor - always was, is now and always will be - for all time.

Accountants get paid squillions to do this.

They get it wrong all the time(BCCCI, Fire Auto and Marine, BP even)and yet they still get paid squillions. Money for old rope, old boy.

Money for old rope.

The head of the audit commission's head should roll, too, for being too soft in the first place - for the team being fobbed off with a quick dirty fix.

It's interesting that the NAO email asking for data to be removed seems to be more concerned with doing so to have the data at a manageable file size than anything to do with only seeing what actually needs to be seen.

These documents show that the sequence of events was not quite as the Chancellor portrayed. The Junior Official would hardly be at Revenue Assistant Grade - as may have been implied??

The letter dated 9 November from the NAO explaining their reasons for requiring the data which was to meet the requirements of International Auditing Standards makes interesting reading in that they say that they usually discuss ways in which the data from which they needed samples could be accessed. There is no indication whether these discussions took place between the appropriate people, although reading between the lines they may not have been. It is clear from the earlier email exchange that they did not really want 25 million records. Given that HMRC could not afford the resources to extract the data in the required format, why didn't someone from the NAO offer to jump on a train to extract the required data? Indeed from an Auditing point of view it would be a guarantee that the data had not been corrupted when copied.

This does not excuse the fact that the data was posted, or that Government departments do not seem to have secure systems in place for electronic transfer of data. That IS down to lack of a decent IT budget.

Well Nick, I did not hear anyone of you ( journoulists or politicians) even gave a thought or asked a question on how much do they think it would have costed to the department to filter the data as required by NOA. It might be diversion from the main topic. In this day and age of techlonogy and databases, it does not cost an arm and a leg for such simple task. However, you will be surprised how much these fat cat consulting companies fleecing from the government. These are same companies which has bungled projects in past and continued to be engaged in a large way, wonder why!!! In this case one might argue that, they have provided what the department has asked for, however, is it not the responsibility of these so called consulting conpanies to provide right advice to the users in maters of security? You should also not be suprised, when this company might be awarded a project to provide secured data from their own system in a CD in a secure manner!! Don't you wonder how billions of pounds are transated over network in the financial services for decades now without such a fiasco!!!!

Hi Nick - putting the entertaining world of politics to one side for a moment and not employing rocket science I cannot help but comment:

Assuming highly confidential information has to be passed between Goverment departments miles apart - why isn't it being transmitted as encrypted data over secure business broadband links or satellite links with minumum cost. Even us plebes have understood 'Secure Servers (SSL)when it comes to placing web transactions. CD's for serious and secure business storage purposes went out with the year dot.

Does anyone else find it amusing that the civil servants who failed to protect the identities of 25 million people have their own identities protected in these emails??

Actually, I dont find it very funny either....

Someone's not telling anything like the truth here. A CD holds about 600MB of data. If this covers the data for 25 million people, there's only about 24 bytes of data per person. That's about 24 characters or numbers per person, enough for a name and little else. Even on two CDs the amount of space isn't enough. What's actually been lost is probably a hard disc drive, or two, with the life history and every recorded detail of the entire population.....

There is something more worrying to me than the fact that these disks may or may not have fallen into the wrong hands. It appears that a relatively junior member of staff could just download the files onto a CD and post it without anyone knowing.

How can we be certain that no staff member has previously downloaded the files and sold them to a criminal gang? Forget the case of the missing CDs, if our data can so easily be taken, how can we be sure that this has not already happened secretly?

Personally I do not think such a system should allow downloads that easily. It should be locked encrypted in the system.

This system was not safe to hold our data. The inquiry should start at the commissioning and design of the system.

It's an interesting set of emails.

First, the files appear to be zipped, presumably with this mythical password. As most technically competent people will know, zip passwords are not secure and can be broken with ease.

Second, if I'm reading it right these zips contain CSV or Excel data (mention of headers), they are not extracts from databases as such. Its a nasty way to deal with data of this size.

Third, they would have been better off to transfer the files over GSI, the Government Secure Intranet. Both emailers are on it and although it would have taken an age given how slow it is, no post or TNT would have been involved. In general you would have expected this data to be hand carried anyway.

Note EDS involvement - they don't have a good record of involvement in government IT projects do they?

Not sure I see what Alistair Darling's doing wrong here. He informed everyone at the right moment - i.e. when he had taken advice when steps were fully in place and the need to inform the public outweighed the risk of mass panic. No doubt someone will enlighten me otherwise. Or for that matter how a Conservative Chancellor would have done any better.

Now I DO have a problem with a public which expects computers to be used for the efficient electronic processing of data - and yet fails to accept any responsibility when it transpires that errors are likewise magnified in their effect. One leads to the other. While we're at it, who's changed their passwords?

We keep demanding info, not always in our own interest. Yesterday James Naughtie, interrogating Darling on the Today programme, was trying to establish what protection was on the discs. Darling wasn't saying, which seems more than a little sensible if you don't want to give too much help to potential identity thieves. Naughtie got the message eventually, but what was he doing pushing the point in the first place?

The answer's simple. The public has an insatiable appetite for scandal.

It keeps you in a job, Nick.

Let's look at this in more detail. The NAO doesn't deal with clerks, they're concerned with the overall performance of the body and only talk to the top management and accountants. This will involve some routine sampling to verify the acounts presented, and as much as possible will be done in London, to save on mission costs (the boss spent the entire budget on himself) and that would have been delegated by the HMRC Mandarin Eloi to one of the Morlocks. Said Morlock, knowing there's no money to be spent on NAO queries, gave the Eloi the disk and told him he didn't have time to winnow the names.
Now this tells us quite a bit more about the said database. A hundred zip files, that's a thousand files. A thousand files isn't a database, it's a junkpile, and probably unusable: I shudder to think what the NAO would have made of that lot. If there IS a management system running it, it's almost certainly system-dependant and irreplicable, and (like the Morlocks' machines in HG Wells' work) unmanaged in practice other than in partial areas where a vague folk memory of what needs doing is maintained as a quasi-religious observance. And in that may perhaps lie the Big Lie - do the disks actually exist, or is this an HMRC attempt to hogwash the NAO by getting them "lost in the post"?

I had a look at the emails and you can hardly read them for all the black ink hiding the names of the fools and incompetents who are responsible for this farce. It's a disgrace that the public who pay the wages of these fools are not allowed to know the names of the people responsible for wasting their money. About time we had genuine transparency in government instead of this sort of ludicrous censorship.

I am amazed that the emails between HMRC and NAO failed to consider the requirements for protective marking of the physical copy of the sensitive data that NAO requested. A copy of the data for just one individual would be marked RESTRICTED; data for 25m would surely require to be protectively marked at a much higher level and require special handling procedures by both HMRC and NAO. Failure to handle protectively marked documents correctly is a serious matter; it seems failure to mark them properly in the first place is not. Someone needs to din it into the heads of senior NAO and HMRC folks that mass personal data is highly sensitive material of national importance subject to the Official Secrets Act, the Data Protection Act and the Human Rights Act. That no-one can be prosecuted for this breach is amazing.

It beggars belief that such a monumental cock-up could happen.

I wouldn't be surprised if the next snippet of news that filtered out is that the password for these CDs is 'password' (or similar).

Surely (someone, someone please confirm this), the IT bods at HMRC have heard of encryption?

I find hard to belief that in the 21st century UK government departments are actually passing extremely sensitive data to each other by courier on CDs and with only password protection.

The sheer stupidity and ineptness of such a practice simply beggars belief.

Why on earth are they not transferring the data in encrypted form over the internet, preferably via a Virtual Private Network (VPN) for extra security?

Secondly, why is the HMRC IT department unable to provide just the data fields required by the NAO? What kind of antiquated computer systems are they running if such a request cannot be met on the grounds of cost?

You've missed the best bit. The documents were zipped. That means they probably have zip passwords. Which means they are encrypted after all. Which means HMRC are too dim to even realise they did encrypt the disks.

The last line about sending safely looks so incongruously formal in comparison to the rest of the letter it could be argued that it's been tweaked at a later date...

also very interested to see the ending of the email addresses for the people who were CC'd...heaven forbid it would be .gov.uk ....

Charles Hardwidge - that will be the programme of delivery that was started on 1 May 1997, has seen some of the greatest per-capita tax revenues ever taken from the British people over that period, and we've still got poorer schools, hospitals and systemic failings in our large departments of state. Hardly for us to congratulate them is it?

If it was possible to burn the contents of that database to CD once, it could have been done many times. It shows a complete lack of understanding of database security fundamentals. I'm a database administrator and the change that they were asking to make to the columns being extracted from the database would have taken ten minutes for anyone to do. Presumably the private contractor that was doing the work had a piecework billing for each extract - hence the 'increased cost to the department' one assumes. Out in the real world, if I get asked to produce an extract from the systems I work on, I just get on and do it. For our tables with client names and other 'level-3' sensitive information, whilst I can formulate the extracts I can't see the data itself - it's encrypted. If the files on these disks had been encrypted it wouldn't have mattered if they'd have been lost, they would have been useless. But by the sounds of it they've sent them in unencrypted format with password-protected zip files (the IT equivalent of tying your bike up with a bit of string, rather than a big shackle lock, and hoping it won't get pinched) and then compounded this by sending the zip file passwords in plain text (leaving a pair of scissors next to the string).

If HMRC have this level of security around one database system then they have them around all similar systems such as PAYE, NI, tax returns, etc - and that means, no matter what the business controls you put round it, any IT guy with half a brain can dump any of this information to disc and no-one will know about it (because, if they've been this lax is setting up the security, you can be sure that the y won't have made it impossible to turn off the audit trail of who has done what!)

Very interesting Nick!, I find it absolutely shocking and disgusting at the complete criminial disregard for proper Data Protection by the HMRC, so my question I have for you Nick is, what is our law enforcement agencies going to do in terms of criminal proceedings on this? or will this be another of those cases, where after we see a long police investigation the CPS just turns round and says 'not enough evidence to prosecute!', simply because it would make Brown & co look tarnished!, my nephew and niece are only two among the 25 million others who at risk from identity fraud now from this fiasco which makes simple parliamentary 'sorrys' offered by the PM & others little justice!, so as I see it those that broke the law should face the music in the law courts, because then and only then Data Protection might be observed with greater care once this high profile example is made!, but either way, one thing is for sure this government will reap the whirlwind of our anger come the next visit to the ballot box!

From my point of view as a professional DBA (that's Database Administrator), the interesting thing about the e-mails is that they tend to indicate that the problem was systemic rather than a single office junior overstepping the mark.

Instead of writing a data extract for the specific data required (or rather having EDS write it - they probably being the sole repository of database knowledge), a copy of the entire live database is sent. This is done purely to save on costs, despite breaking more rules (and laws) than bear thinking about.

The correspondants really do not grasp the full implications of what they are doing - and certainly the technical process involved had obviously not undergone any sort of review (by anyone competant at least), presumeably to keep costs down further.

I have to agree with C E Hardwidge - these emails seem pretty innocuous to me. You start by talking about the 'tone' demonstrating little concern about data protection, but then quote a line directly regarding the email author's concern about the safety of content. Nick, I think you're being pretty petty on this one. And what's wrong with Civil Servants concerned with keeping costs down? Your tone seems to imply some sort of stinginess here - but of course they should be keeping cost down - it's public money! Mistakes were made, and the process was not as robust as it should have been, no doubt, but I think more can be said about the tone of your over zealous textual analysis than the tone of these rather boring, normal emails.

Nick, The question the Government should be answering is how they sanctioned a system so insecure that the files could be copied on to CDs in the first place. Once they can be copied the box is open, and the results are plain to see. As a former Chancellor GB should know this.

The point about saving cost is interesting. Given that governement computer systems are run in-house, it seems probable that the cost being saved does not represent real money, only an internal charge within the civil service. Hence the reference to "cost to the department". In other words it would not have cost the taxpayer a bean to do what the NAO asked, merely resulted in a notional charge going against the Child Benefit department's budget.

Now that it's been established that a Director-grade was responsible for the decision at the root of this problem, might I express my disgust at someone of that level allowing a junior to take this kind of rap for him? Not only did he not give a monkey's about the security of an entire generation, but he didn't even care for the staff who had faithfully executed his instruction to the best of their limited ability. That's pretty serious, and if there are any Unions worth talking about left, something they should be seriously engaged about.

There's no explicit mention of "our privacy and rights", but the letter does ask that the disks be delivered "as safely as possible due to their content." Surely that's clear enough.

Well done Charles E Hardwidge for failing to look at the facts and accepting all the Spin that Brown &c have put out about this. This Government have never focused on delivery, they simply pass laws and hope that things happen.

Keep up the good work Nick - nice to see at least some of the source documents here.

Two things of further concern seem evident from the email exchange now published.
1. The NAO was given a full copy of the Child Benefit database back in March 2007 so this is not the first time that someone has created a copy of the database. Do we know who authorised the previous extract?
2. The NAO concern for the secure delivery of the October data raises the question of whether the cds were delivered in March by unregistered post as well and that this lapse in procedure was noted at the NAO but not thought serious enough to warrant any more than an aside in an email when the data was next to be sent.
The evidence seems to point not to an isolated incident but an accident waiting to happen!

So, in answer to Mr Hardwidge, this does add something: we now know that the files were not encrypted (as many of us in the secure computing community surmised), but merely password protected zip files.

To explain this difference: the first should be secure against almost anything in existence today, the second is less troublesome than today's Sudoko -- even if the emailed password is not intercepted.

The real issue is not whether or not some poor soul evaded the rules, but instead why it was even possible to evade the security policy (i.e. download the whole database) in the first place.

... Were I HMRC, I'd seek to place the blame on EDS for doing such a lousy job of data security. They in turn will blame poor specification by HMRC. All of this happens if you buy an IT system without the relevant internal technical expertise.

My conclusion: if this is blamed on a freak occurence of "rule-breaking", then expect it to happen again and again. If, instead, the problem is correctly placed on poor procurement procedures, and senior management failures (they really should be able to get accurate feedback from "junior staff", who almost certainly do know that this is not the best way to do things), then things might just get better.

Personally, I'm not optimistic anything will be learned: in my part of secure computing (avionics) we have a saying: "Everytime I make a mistake, we can blame the pilot".

One other tip: listen to Ross Anderson; he knows what he's talking about.

"They demonstrate little concern from either the NAO or HMRC about data protection" looks rather daft when the mail says "Please could you ensure that the CDs are delivered to NAO as safely as possible due to their content".

Nick et al

My understanding (from the emails and other reports on various ±«Óãtv channels) is that only the NI Number was required by NAO and that the data wasn't segregated because it was either (a) too inconvenient, or (b) too expensive (I heard a figure of £70,000 quoted), or (c) both. As someone who works with data in a commercial environment every day this seems extraordinary.

If the database had been designed properly then to select only the NI Numbers is a simple exercise - SELECT [NI_NUMBER] FROM [A_DATA_TABLE]. If it couldn't be done that simply then it shows a complete lack of competence in the design and development of the systems.

If the contractor employed by HMRC to run the database is quoting £70,000 then it shows a complete lack of competence (and probably knowledge) amongst the Civil Servants who specified and managed the procurement process to such a degree that huge amounts of tax payers money is being wasted.

My view is that not only are both of the above probably true, but there was also (as Nick states) a complete lack of thought or regard to ensuring that highly sensitive information about the tax paying public (who pay these Civil Servants salaries) is transfered securely.

HMRC was created by Gordon Brown and this massive error sits squarely on his shoulders. By the time Darling took over it was too late!

"They demonstrate little concern from either the NAO or HMRC about data protection" looks rather daft when the mail says "Please could you ensure that the CDs are delivered to NAO as safely as possible due to their content".

Nick,
Love the muckraking and playing to the gallery! You should do it more often. Beats sticking your head in the sand when anybody says or does something that does fit in with your agenda!

Interestingly the civil servants have used black ink on the web site url - why wouldn't they want us to know this?

See pdf page 9

Re my previous post - having read the pdfs more carfeully there is reference to EDS running the extract. EDS is the external contractor that obviously runs this system so potentially there would have been a charge in real money. However there are still two points to be made: (1) it would only be a real charge because the government contracted the work out and (2)nobody bothered to check whether there actually would be a charge, they just left all the sensitive data in in case there would be a cost.

Another point in the pdfs is that, although the NAO asked for the sensitive data to be excluded,they appear to have done so purely to make the file smaller (presumably to make it easier to handle at their end.) The sensitivity of the data and the potential security risk do not seem to have been at issue initially although NAO did later advise HMRC to make sure it was transmitted safely.

The politicians' only thought is to save their own skin. It wasn't me Miss, it was 'im. There seems to be a feeling around that it's more acceptable if a junior did it. Well to the Government yes, but not to us. The fact remains that, whatever the protection afforded by the 'system', the data were lost. It seems to have happened because a human decided not to follow the rules. That is the crucial point. Whatever the rules, someone will always break them. The Government has been warned that the existence of very large databases legitimately accessible by very large numbers of people 'is an accident waiting to happen'. That will always be so, whatever the rules are. The solution is not to amass large databases and to ensure only a few high level people can access them. Better still, don't have them at all!

There's every indication here that the HMRC used winzip to password these CDs. Which means they're more than likely well encrypted, depending on the version used and the length and nature of the password. Sounds like they've oversold their error.

So, in answer to Mr Hardwidge (#7), this does add something: we now know that the files were not encrypted (as many of us in the secure computing community surmised), but merely password protected zip files.

To explain this difference: the first should be secure against almost anything in existence today, the second is less troublesome than today's Sudoko -- even if the emailed password is not intercepted.

The real issue is not whether or not some poor soul evaded the rules, but instead why it was even possible to evade the security policy (i.e. download the whole database) in the first place.

... Were I HMRC, I'd seek to place the blame on EDS for doing such a lousy job of data security. They in turn will blame poor specification by HMRC. All of this happens if you buy an IT system without the relevant internal technical expertise.

My conclusion: if this is blamed on a freak occurence of "rule-breaking", then expect it to happen again and again. If, instead, the problem is correctly placed on poor procurement procedures, and senior management failures (they really should be able to get accurate feedback from "junior staff", who almost certainly do know that this is not the best way to do things), then things might just get better.

Personally, I'm not optimistic anything will be learned: in my part of secure computing (avionics) we have a saying: "Everytime I make a mistake, we blame the pilot".

One other tip: listen to Ross Anderson; he knows what he's talking about.

In response to those wondering about encryption. It all depends on the version of Winzip and the level chosen. Any chain of encryption is only as strong as it's weakest link.

There are several published attacks against the latest version of Winzip. The earlier versions are trivally broken in seconds and offer no protection at all. In light of the fact that the latest version costs money and offers no real advantages I would be surprised if both ends were running the latest version.

Even *if* the latest version were used then the password strength must be considered against a brute force attack. Given that they didn't have the first clue about basic procedures or securty, I'm not expecting a strong password to be in place either.

Adam,
Pretty clearly, if the disks were well encrypted we would all be reassured to hear the Chancellor state it clearly. He hasn't though, has he?

If the CDs were well-encrypted the Government would not have needed to make such a public issue of the data CDs being lost, unless of course the de-cryption key was with the CDs. Don't forget that the RIP Act shows that good encryption can defeat even Government-level attempts to de-crypt.

The censored emails made available and the carefully worded announcements give the impression this fiasco is all the fault or one irresponsible 'junior officer' in HMRC. However, anyone with professional experience in securing data can see we are not being given
the truth. So there is not only an incredible security lapse, but there is also a cover-up.

And finally, Government's misleading announcements have been minimising the long-term consequences of losing these CDs. For anyone who states this is not a severe risk, I
challenge you to put your equivalent details into the public domain.

I was going to suggest that the disks had been hand delivered. But then missing laptops and files carried by hand are also good at disappearing without trace.

Nick, the untold story is that (despite statements to the contrary) NO processes were broken.
(1) There is NO common UK government encryption standard. Not like in the US where they have the AES (American Encryption Standard), or in Germany where they sponsored work on a publicly evaluated version of PGP called GPG. So, in absence of proper encryption tools you cannot fault the junior for not using something proper - it doesn't exist!
(2) Internal government mail USED to be secure, but has suffered like practically everything else from being outsourced. After all it carries other things like paychecks.

Regarding the previous post:

AES is the "Advanced Encryption Standard", not the American Encryption Standard. It's Dutch. Everyone can use it. Everyone can use the "olde encryption standard" Triple-DES. Everyone can use GPG (which is RSA + Triple-DES or AES + user interface).
Everyone can use PGP, which is equivalent to GPG. You can use "openssl" on the command line, too, selecting your symmetric cipher of choice (my company does when encrypting database dumps that go to tape)

Encryption is just a TOOL, and orthogonal to "processes". For example, for exchanges of restricted documents between ESA contractors, one would use the accredited encryption tool CHIASMUS (which is NOT GPG), but there is a book of (admittedly rather high-level) instructions on how to actually handle restricted materials, whose most general expression is given by a document called "BASIC PRINCIPLES AND MINIMUM STANDARDS FOR THE PROTECTION OF CLASSIFIED INFORMATION PRODUCED AND TRANSMITTED IN CONNECTION WITH ESA ACTIVITIES" (Google for details)

There MUST be a common UK document of similar intent. MUST.

I'm an IT professional, but neither a database nor encryption specialist. However, several things stand out here for me:

(1) Technical difficulties: If I had to extract specific records, rather than the whole lot, from any usable database, I would expect the query to take no more or less time than to extract the whole lot. I assume they already had the full dump to hand, and expected to be billed a large amount by EDS for any further extract. I further assume they've downsized, laid-off, demoted or ignored any in-house IT expertise they have, and arranged things so a private company with a track record of late delivery and general cock-up has absolute power over public records. I may be doing them a disservice, but, depressingly, I fear I am probably not. If I had to encrypt sensitive data, I would do so in-house, using a strong encryption package designed for the job. If the data were sensitive, then I wouldn't expect office juniors or un-cleared personnel to have any access to the unencrypted records at all. Nor would private subcontractors, who might employ ANYONE on minimum wage, have access to those data.

(2) Size: 25 million records, compressed with ZIP. Let's assume a compression ratio of 10:1 for typical plaintext with a lot of repetition, we're looking at about 560 bytes per record assuming two 700MB CDs. Since the civil servants, ministers and apologists concerned appear not to know their arses from their elbows where IT is involved, however, we might in fact be talking about two DVDs (at 4.7 or 8.5 GB each) or two of any number of outmoded or esoteric magneto-optical discs, or two hard discs at up to perhaps 750GB each. Or indeed two dinner plates with dymo tape stuck to them. Sticking with CDs, 560 bytes per person is ample for name, address, NI number, bank a/c number, sort code and sundry odds and sods.

(3) Professional DBAs in business are used to dealing with sensitive data via encrypted exports to which mere lowly DBAs don't have keys. The data are stored and handled in encrypted form. It doesn't look like that's the way the govt. or EDS work. Doesn't bode well for the National Identity Database, or the DNA database.

(4) "Encryption". Early password-protected Winzip provides security about as good as handing the info out on flyers in the street and hoping no-one reads them. Technically aware children and clued-up adults routinely "crack" this kind of password protection, in seconds or minutes. More recent cost-option versions of WinZip use "strong" encryption, for which the incentives have to be a bit better to persuade someone to put the hours in. There are vulnerabilities, and rest assured, if the payoff is worth it, the hours will have been put in. Offering to send a password in an email suggests that the civil servants involved here really aren't aware of, or don't think it's important to observe, even very basic common-sense safeguards, so my money's on their using old, free, insecure "password-protected" winzip. It might seem like a good idea to send a password AFTER the discs have been safely received, but the discs could easily be copied in transit, and the released emails have auto-generated footers pointing out that the emails are insecure per-se. Indeed, the ray of hope for the 25 million here is that if they WERE nicked in transit by organised criminals, the criminals would, if they had any sense, simply have copied the discs and then the NAO et al would be none the wiser. So they probably ARE just lost.

(5) If I had to use a single word to summarise the attitude of the UK government towards UK citizens in respect of an individual's rights and security, that word, on the strength not just of this but all other evidence available to me, would be "contempt".

(6) The Cast: Whoever it was in the comments above who wonders who EDS are, they're the IT outsourcing company responsible for a selection of major-league government IT cock-ups. They're good at government IT cock-ups, that's why they keep getting the contracts. I guess someone knows someone and it's all very chummy. The NAO, that would be the agency until recently fronted-up by Sir John Bourn, who, over the last three years, ran up around three hundred thousand pounds in expenses (on top of his hundred and sixty grand per year salary) flying around the world first-class, staying at rather expensive hotels with his wife and dining out at gourmet restaurants while, presumably, keeping a close eye on civil servants' wanton overspending on stuff like, I dunno, IT support maybe. Clearly the stewardship of our hard-earned tax money is taken every bit as seriously as our privacy. (See Private Eye).

When someone breaks the law and admits to it, the police pounce and prosecutes , which improves "Crimes solved" figures. Why are politicians who admit breaking the law free from prosecution? I am sure that all law breakers will confirm that they did so 'in the spirit of the law' which would receive short shift from the police and courts. Admitted guilt does not need an enquiry by fellow MP's.