Operation War Diary launch, 14th January 2014

[Chris_Baker]

The available bit ends when they return to the 56th Division, Mike. You might have a long wait for the rest!

[Guest]

I thought that might be the case. Thanks Chris. Oh well, we've waited 100 years, a few more months won't make a difference. I appreciate all the work that has been done, and it looks like they have done a great job. Patience is a virtue.

Mike

[Roxy]

I'm doing 1 Carmeronians (Scottish Rifles) late 1914 War Diary.

It is quite addictive!

Incidentally, my email from them appeared to indicate that I was number 916,000 or thereabouts - no sure that I believe that.

Roxy

[David_Underdown]

Yes, I should have made that clearer, if a unit transfers to another division (outside those currently available), or potentially if a division transfers to another theatre (eg Italy) then the diaries would end at that point. This is due to the fact that the diaries are organised hierarchically, with subunits under their parent division, and divisions under the relevant theatre of war.

I think the user ids are valid across all the projects Zooniverse run, so that's also probably a total number of Zooniverse users

[Suehat]

I've also signed up & have been tagging. I agree with the others who have said it is adictive! But I'm finding it very enjoyable, nice to be able to do one's bit!

[rflory]

I download 12 artillery war diaries last night (US Pacific time, when most Brits should have been asleep) and eight of the diaries downloaded immediately, while the other four indicated that the "were not available". Three of the "unavailable" diaries were finally available three hours later and the fourth was not available to this morning. It should also be pointed out that the cost of £3.39 is not always for the full war diary but often only a portion. Some of the CRA diaries required four to five separate portion for one year of war diaries, each portion costing that amount.

[David_Underdown]

The diaries have had to be divided into chunks that reduce the overall load on the system (as you can see we perhaps haven't been entirely successful on that front), and we are also required to operate the online service on a cost recovery basis, so we have to try and divided them up in such a way that the costs are properly apportioned.

[spof]

The diaries have had to be divided into chunks that reduce the overall load on the system (as you can see we perhaps haven't been entirely successful on that front), and we are also required to operate the online service on a cost recovery basis, so we have to try and divided them up in such a way that the costs are properly apportioned.

Thanks David.

I had wondered about the changes and you've identified 2 things I had - load on the servers and cost recovery. I always thought that the price of a nearly complete Brigade being the same as, say, a 3 page will was a little out of kilter.

The other things I had thought of was broadband speed as not everyone is on super fast connections and feedback from people only wanting a month or so and having to get 2 or 3 battalions.

[David_Underdown]

Of course previously everything was black and white (being in colour immediately puts file sizes up), and the digitisation was done (I think) from microfilm prepared many years ago. Microfilm you can very easily put through high speed scanners and process a whole film into digital images in short order. For this exercise we have gone right back to the original boxes, so every page is turned by hand. Incidentally, some of the reports have suggested that the volunteers who sorted the material were also involved in the scanning. In fact the scanning was carried out by specialist contractors, all on site at at Kew (official records are not allowed to leave the building for scanning). While we do have an in-house scanning team, there is sufficient to keep them busy anyway, so we work with a range of specialist scanning firms for these larger scale projects.

[HeatherC]

Regarding the Zooniverse diary tagging project. I've decided to stop doing it after a couple of days. It appears the oprganisers cannot comprehend the difference between having people tag a bunch of random galaxies and tagging a STORY which requires some continuity and some knowledge. The response to suggestions, comments and ultimately complaints on the site's discussion boards has been rubbish. I think during the whole of yesterday (the second full day of the project) there were about 4 moderator responses. Absolutely woeful. The number of times that useful and interesting data simply has to be tagged as "other" is incredibly frustrating and it also seems that many of the volunteers are struggling to get the site software to work correctly and are wasting huge amounts of time re-doing stuff or just giving up altogether.

The lack of continuity was demonstrated nicely for me this morning. I've been doing 1 Glosters in 1916, which had a personal interest for me as I have a relative who was killed in High Wood in Sept 1916. I have of course seen large chunks of the diary before as I paid for them in the old fashioned way! Because I have some knowledge of the Unit and the locations I think there is an advantage to me doing this diary as I will miss less and understand more than someone who does not have that knowledge. I stopped last night at the end of September 1916,. I log in this morning to be greeted with the words "thank you for completing the diary of 1 Bn Gloucestershire Regt, please select another diary". It turns out they allocate each page as a separate chunk of data to anyone who comes along and picks it and so the continuity of the story and the knowledge I had gained of names and places is totally gone in the rest of the tagging of that diary, which I presume was done by someone else who had no idea that the Bn had just been decimated in the High Wood attack.

So it seems the people running this are more interested in their statistics of the amount of data completed than in actually getting it right, or in having knowledgeable people who actually have an interest in the story rather than just in tagging data. Great pity.

[HeatherC]

Since I posted the above they have started to make some sense of the various suggestions for improvement. However the issue that each page is treated as a totally spearate piece of data and you may not get to see the conitnuity of the story if you go away from tagging for any length of time still remains so bear it in mind if you get involved in this.

[SilverFox100]

Have been involved in his operation now for a few days and find it of great interest. It gives one a chance to look at various of the diaries of interest to one, to do as much work on that as you want, then take another one. Obviously deciphering some of the handwriting can be difficult but I am finding that ok and am really enjoying doing it. However, these diaries are getting completed very fast, when I started some days ag there were 5 full pages of about 20 or so per page and this is now down to just less than two pages of diaries, so this is proving to be very popular. If you are interested get involved quickly. I don't know if any more are due to be released but I hope so. Such a wide range of diaries have been on offer.

Mike

[Guest]

The launch of these diaries is simply wonderful. TNA should be applauded . It is a truly monumental effort. I suspect by making the diaries more accessible it will open up new avenues of research for many who were not able to make the journey to the Nirvana that is TNA.

TNA appears to have realised they were selling the diaries at ridiculously low prices. The latest release has certainly sliced the loaf of data into much thinner slices, so we get a lot less for £3.36 (or £0.00 if downloading at TNA). There still appears to be zero consistency. I can get the War Diary for the whole of the War for some battalions, yet only get 3 months' worth of other battalions for the same £3.36 Scale that up for 52 months and it makes £58.24 which might make some eyes water..... Personally I don't mind paying more as the diary archives are simply a world class asset and must cost a fortune to maintain.

If anyone from TNA is listening - congratulations on a great job, but please, please help us help you and get some kind of standardised packaging and pricing. It's not difficult. A MIC for £3.36 and 300 pages of a War Diary for the same price simply does not make any sense for the supplier (TNA*) or the consumer (us).

MG

[Guest]

" If anyone from TNA is listening - congratulations on a great job, but please, please help us help you and get some kind of standardised packaging and pricing. . "

I agree on a job well done, and wonderful as you say to be able to view them, without the pilgrimage to Kew. They could standardise packaging and pricing, the same way the Canadians and Australians have?

Mike

[gilly100]

Great initiative and a big job no doubt. It also only highlights what a great job has been done in Australia already with our records. I personally believe the diaries should be available VERY cheaply or free, and that though taxes people already pay, TNA should be properly funded.

My two cents worth on YOUR history assets, so happy to butt out. Hope the job gets done.

Ian

[Guest]

I suppose there are many more British Diaries, than Canadian or Australian, so a bigger job. £3.50 ain't bad, but it adds up.

Mike

[gilly100]

Yes, totally understand that Mike and also trying to rationalize a price based on the quantity of material. My only point is that these records are a national resource that taxpayers already fund I would have thought. I guess it just depends where the government spends your pennies! All, the same, a great thing to get done over time and making access all the easier for those that cannot get to where the records are currently held.I have bought a few Brit diaries already and certainly do not begrudge the price.

Cheers

Ian

[spof]

Have been involved in his operation now for a few days and find it of great interest. It gives one a chance to look at various of the diaries of interest to one, to do as much work on that as you want, then take another one. Obviously deciphering some of the handwriting can be difficult but I am finding that ok and am really enjoying doing it. However, these diaries are getting completed very fast, when I started some days ag there were 5 full pages of about 20 or so per page and this is now down to just less than two pages of diaries, so this is proving to be very popular. If you are interested get involved quickly. I don't know if any more are due to be released but I hope so. Such a wide range of diaries have been on offer.

Mike

Mike

There are plenty of diaries still to go as there are 10 full divisions which have been digitised just in the initial batch. Others will be online later this year so no need to worry about missing out.

Glen

[David_Underdown]

On a slightly different tack, I note that there are a few challenges ahead: I have found chunks of diaries in the wrong place, pages upside-down, pages out of order (lots) - and my favourite so far, the 2nd Bn RWF War Diary for 1914 (WO 95/1356-1) has duplicated 83 pages. Doubtless they will be all ironed out eventually. My least favourite is a diary with less than 50 pages. It makes the Whole 2nd Mtd Div War Diary (14 regiments and five brigades) look like a steal.

TNA appears to have realised they were selling the diaries at ridiculously low prices. The latest release has certainly sliced the loaf of data into much thinner slices, so we get a lot less for £3.36 (or £0.00 if downloading at TNA). There still appears to be zero consistency. I can get the War Diary for the whole of the War for some battalions, yet only get 3 months' worth of other battalions for the same £3.36 Scale that up for 52 months and it makes £58.24 which might make some eyes water..... Personally I don't mind paying more as the diary archives are simply a world class asset and must cost a fortune to maintain.

If anyone from TNA is listening - congratulations on a great job, but please, please help us help you and get some kind of standardised packaging and pricing. It's not difficult. A MIC for £3.36 and 300 pages of a War Diary for the same price simply does not make any sense for the supplier (TNA*) or the consumer (us).

MG

* I understand TNA is forced to operate on a break-even basis by the crazy Govt (and previous crazy Govts) and that there is an ongoing debate relating to pricing and packaging of products. The current Govt limitation is a crying shame really as it should be able to make a profit to reinvest in its digitisation and infrastructure to broaden access.

We're listening: please report specific issues with downloads via the Discovery helpdesk (I've passed on the issue WO 95/1356-1 to appropriate colleagues).

The current fees order is a fairly blunt object, we can either make records available for free (which essentially occurs where digitisation has been grant funded by some other body), or we have to charge £3.36. This (and the general principle of cost recovery) applies only to records made available on our own website, where licensing deals are done instead, then the cost of digitisation is born by the licensee, and they pay a royalty to The National Archives based on usage. That money is re-invested in other projects.

In addition to the simple cost of scanning each page, then there is of course a cost associated with transcribing/indexing the information to make it searchable. In the case of the war diaries, this is relatively low: there's been some recataloguing when things are broken up into individualy downloadable items but that's not too much of an overhead. In other cases, such as medal cards, that's obviously a much higher cost, and it's not really obvious what else you could give people other than the MIC they want (of course in fact you generally get six at a time).

We are funded for the Kew operation, and for providing the basic catalogue information. The Treasury still views online publishing of the records themselves as a value-added service, and hence cost recovery is applied.

[David_Underdown]

Martin, is it actually WO 95/1365/3 you mean?

[Chris_Baker]

The mathematics of "cost recovery" are interesting. I assume that the £3.36 includes
"a" - an element which is the cost of storage and serving up the document to the customer, and
"b" - an element which is recovery of the funds already spent on digitisation.

The language suggests there is no "c", a normal commercial profit margin built into the price.

At some point, after X number of downloads multiplied by "b", the up-front expenditure is fully recovered.

Does the price to the customer then drop so it represents only "a"? I await that day with great anticipation! And what would X have to be?

[David_Underdown]

It's not quite that straightforward (and this is to the best of my understanding, rather than an "official" answer). The problem is we know there will be a fairly large proportion of files (generally, the war diaries might have a slightly different usage pattern) which are simply never downloaded (and a very small number are downloaded multiple times). So in practice the costing is averaged across all charged-for online material.

[Stephen Nulty]

We're listening: please report specific issues with downloads via the Discovery helpdesk (I've passed on the issue WO 95/1356-1 to appropriate colleagues).

It's worth adding here the point made by Promenade on the other War Diary thread that the SLA for replying to queries is 10 days. Which, let's face it, is very poor.

And in those 10 days TNA have my princely sum of £3.36 which they took by telling me that a diary was something other than it actually is.

[Guest]

Martin, is it actually WO 95/1365/3 you mean?

Yes. It is the 2nd Bn Royal Welsh Fusiliers War Diary. My typo on the WO ref stem. ... pages 3 to 83 are repeated on pages 114 to 194....so 80 pages, not 83.

David - I am not complaining. I am delighted with the material and with 1.5 million pages and humans involved there will always be errors....but I think the layperson will struggle finding some of the pages. I have downloaded about 40 war diaries in the past week and of the dozen I have indexed so far have only found one without a pagination error. Most have multiple errors. It is a small samlpe but I am indexing every page of about 100 diaries at the moment - a rather dull exercise but necessary for my research purposes. I am keeping a record and if it is of use I will email it to TNA when done so the team can sort it out. There are simply too many errors to list here and its probably not the right place to do it, but given the GWF membership has a natural bent for the diaries I thought it worth flagging.... There are dozens of errors - mostly pages in completely the wrong order or completely random unrelated material bang in the middle of a diary.

One example from some indexing I did yesterday - WO 95/1351/1 as an example (2nd Bn Worcestershire Regt)...

1. The diary is supposed to start with Aug 1914.....but 20 of the first 21 pages cover Oct 1914, and are then followed by Aug 1914 i.e the whole of October is out of sequence. I think Stacke was to blame (Official historian of Worcestershire Regt) as it mostly relates to his eulogy on Gheluvelt and some of his correspondence is included. It suggests the mess was caused by some of the Official historians and Regimental historians. Green mention the chaos of file maintenance in his book "Writing the Great War".

2. Pages 36-51 are a printed account of the action at Gheluvelt which was on 31st Oct, so should immediately follow the October diary. It is 14 pages adrift of the Oct diary (see above) but at least it follows September.

3. Page 36 is a duplication of page 11.

4. Page 70 is the cover page for July 1915 and should be the Cover page for Jan 1915.... only 6 months out of sequence and quite confusing.

5. page 71 is the Cover page for Feb 1915 i.e also in completely the wrong location. it should be between page 76 and 77 where Jan ends and Feb starts

This is a rather messy package. .

This is fairly typical; The 2nd R Sussex Diary -page 16 and 17 is a Cover page plus a single page account from a Northants officer and completely unrelated to the R Sussex activity on the single day in question (25th Aug 14). It. randomly appears between 2nd Bn R Suusex Regt diary for Aug and Sep...and doesn't appear in the Northants' diary. Somehow it walked across the Brigade and found a new home. Again I suspect misplaced by the Official Historians.

I will soon have completed indexing over 100 of the BEF diaries and will forward it when done if you think it will be of use as it will be too time-consuming to flag each one as they arise.

MG

[David_Underdown]

Martin, thanks for the detailed description, I think a single report would probably be most useful, probably easiest to email direct to DigitalDownloads@nationalarchives.gsi.gov.uk

Given the amount of resorting of the paper done in advance, and subsequent checking of images, I am rather surprised that so much seems to have got muddled. Given that there is more to come, it will probably be useful to review the experience with this first batch to try to improve subsequent ones.