IWM Lives of the Great War Project - reservations

[keithmroberts]

Luke has made it clear that comments here will be considered. Some concerns might not be met, others will be, when the final version of the user agreement is released. We can't alter the financial structure of a company like Bightsolid, but I'm sure that comments here will influence some of the detail of the implementation. Overall, with others, I see this as a very positive project, and I don't think that Luke and his colleagues will be too concerned about the issues raised - in that, they would rather have them aired now, while some of them can be addressed.

I'm grateful for the constructive replies that we have had, and the clear evidence that the team running the project are thinking about the points raised. Some things clearly won't change - links to copyright images from MIC's or the census will inevitably attract costs. They do now if we view them from home. Luke has equally made it clear that uploaded information from others will not be in that position.

All that I would ask, now, is that members read the FAQ sections of the present site, as well as the comments and replies in this topic before just repeating queries that are covered clearly, or that have had a full reply. Some issues will remain open for concern, and there may be some new ones identified, but please, let us avoid getting into a pattern of just repeating the same questions. They have been read. One of the strengths of the GWF is the skill tha members have applied to their research, so let us show the same skill in analysing the information on the website, and in the FAQ sections, as well as in Luke's replies.

I think this project will seize the imagination of a much wider audience than the GWF, and I hope that our comments will help it to do so more effectively. I look forward to feeding in my research over the next few years.

Actually, that does give rise to a question for the IWM that I don't think is covered at present -

When the anniversary of the Armistice brings centenary events to an end - will we, and by that I mean the wider we, continue to be able to add information to the database?

Keith

Keith

[lukesmith]

Hello,

I'd like to echo a point that has been made by Kate:

...the IWM are engaging exclusively with the GWF. They are not conducting public exchanges anywhere else. That is the mark of the high regard they have for our members and standard of debate.

That was why we asked the forum to allow us to announce this project here and why we are part of this conversation. And it remains the case. We expected that there would be some concerns and reservations and we are keen to have them aired as early as possible. This will allow us to adjust our plans accordingly in some cases, or at least to develop clear responses to all reasonable concerns.

On any online forum and particularly on one where people have a significant personal stake in the topic, the temperature will sometimes increase. Most things raised in this thread have been based on valid and relevant concerns. We will respond to all of them when we are able -- indeed other forum users have already clarified some points of fact and set out IWM's position (thank you!). The main exception to this is the accusation of intentions of 'fiddling' -- but that was a single comment on a forum with thousands of active users. We won't be put off the GWF by that!

Also, I'd like to reiterate our thanks to the many people who have stated their support for the project -- we appreciate it and we are committed to creating something worthy of that support.

I'm going to be away from computers until early next week (lucky me!) so won't be able to contribute further until then.

many thanks,

Luke (Smith)

IWM

[Andy Wade]

Just to clarify a point about war diaries and 'ownership' of transcriptions:

If I spend many hours transcribing a war diary (or any other hand written work, like a soldier's letter home) into text on a computer which can then be copied and pasted (which has a lot of value over and above an image of a page of the war diary) then it is indeed my own work and mine to do what I wish with (it is copyright to me in effect). If I choose to give copyright of this transcription to the IWM for their database by way of an accession record which allows this, then I would like this contribution to be recognised in that the IWM then gives free access to anyone who wants a copy and that I can also access freely other war diaries and other items transcribed by others in a similar way.

My understanding is that this is exactly what will be happening and the opportunities are huge for all of us as researchers.

What we as 'expert' individuals can do is the transcription of things like war diaries and we can also tap into the vast knowledge of this forum to iron out words we cannot recognise (instead of just guessing them) and even giving the comments some context. This is something unique and this is just one aspect that the GWF can offer the IWM, which is why they are here. If there are a few queries and worries then being able to answer them here is another great opportunity to get us all on board in support of it after open discussion. I also support the naysayers because they are asking the difficult questions before the project is rolled out and this will help the IWM to 'get it right on the day'.

I am sure that 'my' own project will be adding to the IWM's database (in bits over the coming years) as it will allow our work to last forever, which is something we cannot do on our own except in a printed form in our own local studies library and archive. That will be accessible only to people who can get there to read it. Our website is as transitory as any amateur website. Even with 'future proofing' we can only view it as 'here today and gone tomorrow'. I would hope that the IWM would be here forever. There is also the issue of our our own local library's future and as we have all seen any of them can be closed through a series of financial cuts in a very small time, so even the printed form of our archive isn't guaranteed a lifetime of access.

I think that some of the concerns (certainly my own) come from the fact that some government supported databases are free (like the CWGC) and others are pay per view (like the NA). It is inconsistencies such as this and the increasing involvement of commercial interests that cause the problems. I think that as long as the commercial entities are 'ring fenced' into their own areas and not allowed to sell freely given information then we'll all be happy with that. It would appear that this is exactly what the IWM is aiming for and I applaud their efforts to do so.

[Tom Lang]

The main exception to this is the accusation of intentions of 'fiddling' -- but that was a single comment on a forum with thousands of active users.

Luke (Smith)

IWM

Luke,

Please show when and where there is "the accusation of intentions of 'fiddling'".

I have previously explained my use of the term, and I have said that my use was not as a slur on the staff of IWM.

I ask you not to shoot the messenger, but to try to shoot the message.

Kindest Regards,

Tom.

[yelob]

Not so much a reservation perhaps but a comment based on recent first hand experience-there is a generation already in existence who really are not interested in history unless its on youtube and no longer than a minute long. My reservation would be, therefore, who is this project for and is there a demand for it outside the narrow(?) confines of enthusiasts? I would like to be wrong, but it may be a mistake to assume that our collective enthusiasm is indicative of the younger and future generations who this project is potentially for.

[Norrette]

Having been prompted by my online diary that I’d better get cracking on my WD transcriptions before I miss the centenary, I am restarting my activity in GWF. Naturally, I’ve read through this relevant post, so I apologise for being a bit late to the party.

I can see the project from both points of view. I too would like to think that what research I have completed, albeit in a small way, is kept in perpetuity. Something on the lines of what Marc Leroux did for the Canadian men is a good start. The official Canadian Vets website is less comprehensive regarding details of each soldier, but is more likely to be still around by the 120^th anniversary. Both of these provide free access to all. I see that LAC are also supporters of this project and I hope that access to their attestation papers remains free.

At one point, I thought about putting an amount aside in my will for ‘rent’ of a website that might keep my data. But I wonder how long would that last? Regarding everyone’s Great War research, unless we get together and do something ourselves – waiting for a free access repository is fruitless.

However, I do share some of the concerns about commercial involvement. Brightsolid do appear to have their finger in quite a few pies and while they are currently classed as a small limited company (a good thing) – the noises from their press releases sound like an enterprise shaping itself up for a public flotation. Will a change in ownership change their quality? No doubt, the IWM representatives on here will be taking our concerns back. I think we all agree - it’s all in the language and terms like ‘platform’ need to be defined.

Of course, as a supporter GWF will no doubt grow in popularity during the centenary – hope we can take the pressure on the site?

[lukesmith]

Hello npm,

Welcome to the conversation. In relation to the following points...

At one point, I thought about putting an amount aside in my will for ‘rent’ of a website that might keep my data. But I wonder how long would that last? Regarding everyone’s Great War research, unless we get together and do something ourselves – waiting for a free access repository is fruitless.

We are committed to building a permanent digital memorial, so your contributions will be part of a legacy for future generations. We state this in various ways in our FAQs, e.g.:

http://www.livesofth...ns.php?sec=1#q1

The permanent digital memorial will be free to access. We do state this in our FAQs, but we could perhaps make a more explicit statement on this:

http://www.livesofth...s.php?sec=2#q13

The core life stories will be available, freely, forever. Records and images that are already free will be free to access, as will IWM records. Other digitised records are only available commercially and can currently be accessed via a payment from various providers (e.g. 1911 Census, service records, etc.). Those records cannot be made free to access as they are bound by commercial contracts with their rights holders. Browsers of the system can view the core life story data without viewing these commercial records. Those who wish to view them within Lives of the First World War can pay a subscription to do so.

many thanks,

Luke (Smith)

IWM

[Chris_Baker]

Luke, a question on permanence. What provision are you making for future developments in technology? This is obviously a bigger issue than one just affecting "Lives", but I am sure you have considered it. What happens when HMTL/XML are as old hat and as impossible to maintain as Lotus 123 or FORTRAN?

[David_Underdown]

Luke will be better able to answer the specifics for this project, but with my professional digital preservation hat on I'd say that HTML/XML would generally be regarded as very low-risk. At the end of the day they are simple text files, and the very nature of XML is that it is very easy to transform into something else (and in fact it may well be that the data is actually stored in some other structure altogether, and the rendering is done on the fly - in fact I'd rather hope that linked open data principles are being sued).

[lukesmith]

Hi Chris and David,

I agree with both of you -- that requires an explanation!

Chris, I agree that we don't want to get stuck with an outdated technology. The aim is the preserve the information, not in a specific data format. We are not going to make that technical decision until towards the end of the centenary when we plan the move into 'permanent memorial' mode (i.e. browsable, but no longer 'active'). Clearly, if lots of activity is still taking place we will defer that moment as long as practical.

David, I also agree with you in that XML/HTML are 'open', text based formats and human readable (up to a point). But new formats and standards may have emerged in 5 or 6 years time, so we will keep our minds open until we have to make that decision. I would hope that the data would be available in such a way that others could port it to new formats, etc. But it is not possible to commit to that level of detail at this stage.

Both points gratefully received.

many thanks,

Luke (Smith)

IWM

[David_Underdown]

Actually, on reflection, the more pressing challenge is likely to be the stability of URLs within external resouces: i.e. when you've linked someone to a census result (or whatever), how sure can you be that the link will still work some period of time later?

[spof]

David

I get the impression this will be a "self contained" system and external links will be discouraged. Assuming census records aren't in the basic details that will be loaded for each man, adding these details will involve transcribing them. This will negate the issue you mention. It won't be much fun if someone has got census records for a whole battalion but for the majority of people it will 2 grandfathers or 4 great-grandfathers and his siblings. I don't think we are at too many great-great-grandfathers yet.

Personally, I'd be more concerned about binary items such as JPGs, PDFs and DOC/DOCX and what happens if those formats become obsolete.

Glen

[lukesmith]

Hello David and Glen,

Good points on stable URL/URIs. We will ensure that these are stable and permanent. If, for example, someone creates a website linking to all the life stories they are interested in, those links will remain valid and accurate. This is important as it provide a fixed point for referencing these people externally. The system will be open in that sense, and we will actively encourage people to make creative uses of that facility.

many thanks,

Luke (Smith)

IWM

[David_Underdown]

I was thinking more of Lives... linking to external resources, rather than vice versa - people on this forum have tried to create links to parts of the 1911 Census where various units were enumerated on their overseas stations, occasionally the census companies have changed their links meaning the work has to be done again.

[lukesmith]

I was thinking more of Lives... linking to external resources, rather than vice versa - people on this forum have tried to create links to parts of the 1911 Census where various units were enumerated on their overseas stations, occasionally the census companies have changed their links meaning the work has to be done again.

That certainly is a problem and we'll be doing our bit by committing to a permanent URL structure -- i.e. setting a good example. We'll certainly encourage others to do the same. Do you have any ideas about how that problem might be tackled?

many thanks,

Luke.

[David_Underdown]

Anyone who can solve the link rot issue will probably get at least a PhD out of it!

So far as the big partners go, engage with them now and explore what the situation is (it may be, for example, that the link to an initial transcript page is stable, while the links to images are not: Ancestry for example have a "save to shoebox" function so you don't have to search for the same records over and over again, so I'd have thought those links must be stable). TNA Disovery links should be stable, and if something does get broken, you should at least be automatically redirected to an archived version in the UK Government Web Archive. Links to other resources (eg people's existing research projects, assorted newspaper archives etc) are inevitably more problematic, to some extent you can mitigate by using the process of adding references and encouraging people to add additional metadata when they add the reference so there's more than a bare url available for trying to refind the content if it goes missing. Also, engage with the web archiving folk at the British Library (and their Australian, New Zealand, Canadian etc counterparts), "Lives..." could be a good crawl starting point for them as it will bring together a nicely themed set of links (and the growth of the site could in its own right be an interesting topic for researchers in future).

[lukesmith]

Morning David,

Thank you for these practical suggestions. We have engaged with some of the relevant organisations -- but I will admit that this is an area we haven't focussed on hugely yet. However, it is very important, particularly for the long term legacy of the project. I may get back to you for more ideas on this in due course.

best,

Luke (Smith)

IWM

[Seadog]

Memories of the time when the CWGC changed all the URLs on the database of the fallen without any reference to the multitude of users resulting in invalid links to individual records. Mind you forum members ensured that things were put right ASAP!.

Norman

[Phil Wood]

That certainly is a problem and we'll be doing our bit by committing to a permanent URL structure -- i.e. setting a good example. We'll certainly encourage others to do the same. Do you have any ideas about how that problem might be tackled?

many thanks,

Luke.

I had this problem with the CWGC when they revamped their search engine - all my links to men/women on their site (about 2,000) disappeared overnight. I was able to reprogramme my site to modify most of the links and get them working - but it took a few weeks and several emails to the CWGC before they modified their programming to correct a simple issue that arose when they created their URLs before every link was reestablished.

Issues such as this will require effort over the years - the site will never be maintenance-free unless it is totally self contained. Even then it will eventually become obsolete and unusable when, for instance, a scripting language or graphics format used ceases to be supported by your average user's machine.

The only way to ensure that external links will always work it to follow them and store the result - like google cache. However, this could lead to copyright issues as you would be storing other peoples stuff. Perhaps you could link the the BL web archive to follow dead links to UK sites??

I guess the real answer is to encourage users not to rely on links to make the life story readable - and to clarify what is being linked to so a reader can find the information through another route if the link dies.

[Seadog]

Perhaps I am taking a too simplistic view of the concept of “referenced sources” as referred to below, but to ensure that such source links will not as it were disappear if and when the URL is changed would it not be better to allow the inclusion of images of the actual source i.e. a soldiers record from Ancestry to be included as part of the details posted by the researcher and not the link to the record. I assume that for a link to be posted as in the example above the individual will have to be a subscriber to such pay-sites as Ancestry so what is to stop the poster uploading the appropriate image of supporting documents etc. I also understand that when such links are included in the posted details then a charge will be made for viewer to see them. As I say a possible over-simplistic view of this particular aspect of the project.

Quote:

To ensure that each life story profile is as accurate as possible users will be asked to show the source of their information. Providing a reference enables other users to check that source for themselves.

Norman

[Phil Wood]

Perhaps I am taking a too simplistic view of the concept of “referenced sources” as referred to below, but to ensure that such source links will not as it were disappear if and when the URL is changed would it not be better to allow the inclusion of images of the actual source i.e. a soldiers record from Ancestry to be included as part of the details posted by the researcher and not the link to the record. I assume that for a link to be posted as in the example above the individual will have to be a subscriber to such pay-sites as Ancestry so what is to stop the poster uploading the appropriate image of supporting documents etc. I also understand that when such links are included in the posted details then a charge will be made for viewer to see them. As I say a possible over-simplistic view of this particular aspect of the project.

Quote:

To ensure that each life story profile is as accurate as possible users will be asked to show the source of their information. Providing a reference enables other users to check that source for themselves.

Norman

I assume you mean that the image sourced from, for instance, Ancestry would appear on the Lives website? But only if you were a subscriber to Ancestry or paid for it in some pay-per-view manner?

I guess this would require a pretty close relationship with the image providers in order to link the Live site to their subscription/payment system in order to decide whether to display the image (which could not be held in the Lives website in the first place without resolving the copyright issues). I guess it could work, but what if they get bought out by a rival and their whole system changes - and the Lives website is in tick-over mode without the support necessary for reprogramming? Much easier to simply link to the commercial site and let them handle the payment side of things.

And we can always hope that, in the future, the PRO images will become available free-to-view or via alternative sites that the user of the Lives site might prefer to use. Even today census images are available from two sites - if I subscribe to one and not the other I would obviously prefer to look at an image from the one I am subscribed to rather than follow a link to the other. Hence my suggestion that authors be encouraged to state what the link is about rather than simply cited as evidence. For instance I could write John Smith's father Eric was a grocer where the Eric link linked to the 1911 census and was cited as anonymous evidence about Eric. Or I could write John Smith's father Eric (shown in 1911 census) was a grocer the latter may not be so elegant but it tells the reader where the evidence is and allows them to decide for themselves where to access it and is still meaningful even if the link is dead.

This also raises an issue regarding Brightsolid's involvement - will they push their commercial resources ahead of rivals? Will it be equally simple to link to census information on Ancestry as to FindMyPast?

[bedfordyeoman]

As a number of members know, I have been researching the Bedfordshire Yeomanry for many years. I have a huge detailed database of information, extending to over 2,000 men. I have also carried out much research into the Bedfordshire Regiment and have information extending to about 12,000 men, some of it detailed, some not. My databases are exactly what the IWM is looking for since I draw together all the known public sources of information relating to individuals but add to them other sources such as newspapers, family accounts, known medals etc etc.

I've lost count of the hours that I've put in. Much of it was pre-Ancestry et al, so it's been overtaken to a large degree. The amount of detail that is available to the public beyond the available online sources, however, is usually very small. Typically, you may find these details for a soldier: (a) MIC ( Service/pension record © medal rolls (d) AVL (e) SDGW/CWGC entry if dead (f) BMD (g) London Gazette. Beyond that, and you really are relying on family and newspaper accounts or local histories. As you'll appreciate, much of this now involves subscription or other payments if you want immediate/easy access

It would be open to me to fill in the 'gaps' between the online sources with information that I have been willing to make available to enquirers free of charge to date.I understand that others are ready and willing to make a profit from information but I have a reservation that I am effectively supporting and furthering the aims of those commercial concerns. And that the open access to data may not be available free after 2018.

I also make the point that it is not possible to identify many men who served in WW1. I adopt a very rigorous procedure when identifying individuals and, dealing with a regiment that covered a very rural area,where people knew each other, I have only managed 70% identification of 80% of the men who served. I venture to suggest that in the towns and cities, the identification rate is likely to fall below 50%.

I also make the further point that many regiments are under-represented in terms of service/pension records. Try finding, for example, a WO363 record for a Beds Yeoman other than one who transferred to another regiment - don't spend much time on it because the answer is that you won't find one (that's over 1000 men with no record of service). I am currently of the view, therefore, that the project will, in many cases, amount to no more than an MIC and, in the event of death, a CWGC/SDGW entry.

I await further details as the project is rolled out but, at the moment, I'm thinking that I probably won't participate.

Best

David

[David_Underdown]

The great thing (potentially) about this project is the breadth of sources that can be drawn on, and the contacts between people researching the same people from different angles, potentially far greater than any single researcher could otherwise hope to manage. For example, in the sources I've been looking at, there are 8 men identified as being Bedfordshire Yeomanry, but without these sorts of conversations I wouldn't know there was anyone who would be particularly interested in them (DM me if you want details).

Almost certainly there will be some men where it will be impossible to progress beyond a MIC (and of course, if they didn't serve overseas, even that may not exist). Equally, local news reports etc do contain many errors, but by combining enough of them, and many bits of circumstantial evidence, you can build up a surprisingly accurate picture (essentially, this is how the things like PRISM that are currently in the news work, identify ltos of small deatils, and the big patterns start to emerge). Sometimes it may be a case of Sherlock Holmes style, eliminating the impossible, and that means there's only one possible identification left.

One thing that would be useful is in the "Lives..." site would a means of identifying the possibility of two men actually being identical, without absolutely asserting it. For instance, on the memorials I've been looking at was a man named Charles Herbert Varo. Quite an unusual name, and Soldiers Died allowed us to confirm the identification between the man shown as being from Monk's Eleigh in Suffolk and Burstow in Suffolk. Then he also popped up on a memorial in Bledlow Church, Buckinghamshire. The name suggested it was the same person, but it was only when I found reports that Varo had moved from Suffolk to Surrey because he was gardener to a clergyman who moved between the two parishes, and then further reports that showed that the clergyman moved from Burstow to Bledlow in 1917 that the link to Bledlow became clear.

ONe last thought (which otehrs have essentially mentioned upthread) - what's the point of all the research you've done unless it's available to others? And as soon as you make it available, there's always the chance that someone else will be make to profit from it in someway - even if that's only publishing a book that uses your work to improve the information on one man.

[E Wilcock]

I've lost count of the hours that I've put in.

It would be open to me to fill in the 'gaps' between the online sources with information that I have been willing to make available to enquirers free of charge to date.I understand that others are ready and willing to make a profit from information but I have a reservation that I am effectively supporting and furthering the aims of those commercial concerns.

David/ Bedford Yeomen makes a point that is really important to me. But which is not specific to the IWM anniversary project.

My husband and I have no funding for the work we have done on RFA 243 Brigade. We have willingly dedicated our time, paid for subscriptions to on line records and put our family papers on the web.

But the Artillery Museum and Archive at Woolwich (Fire Power) is run on commercial lines and it would cost my husband and me £20 (£10 each) to use the archive just for one day. Those able to afford Ph D fees and affiliate to a University, get free entry.

For this reason we have never been to the Artillery Museum archive. It is a matter of principle. Authors of academic articles are not paid for their published work.Serious researchers should have free access to source material.

But pricing volunteer researchers out of archives is not something ever done by the IWM, to whom we are very grateful and where we have had help and free access over many years of academic research.

It would be wrong for this thread to associate problems faced by independent and volunteer researchers, with the IWM which is blameless.

[John_Hartley]

I venture to suggest that in the towns and cities, the identification rate is likely to fall below 50%.

Certainly there will be an issue in urban areas in making sure a correct identification has been made. A past research project of mine related to the men commemorated on Stockport's war memorials. Even with the knowledge that a man was dead, there are still many (perhaps around 10% - I've never counted them) that I simply cannot ID. This is mainly due to them having relatively common names. And I've had a couple of cases where I have been certain I had made a correct ID, only to be later contacted by a descendent telling me I'd made a mistake.

The problem will be multiplied when the project comes to deal with the much greater numbers of men who survived. Have to say, I am unsure how the project can ensure rigour has been applied to get the right man.