Live page, screenshot generated on Sep 9, 2016

First embedded tweet - live web

Archive-It, captured on Aug 18, 2016

Embedded tweet as displayed in Archive-It

Webrecorder.io, viewed Sep 29, 2016

archive.is, captured Sep 9, 2016

WARCreate, captured Sep 9, 2016, replayed in webarchiveplayer

WARCreate capture replayed in webarchiveplayer, with tweets missing

• Archive-It (Heritrix/Wayback) - archives the HTML returned in the HTTP response and JavaScript loaded from the HTML
• Webrecorder.io - archives the HTML returned in the HTTP response, JavaScript loaded from the HTML, and JavaScript loaded after execution in the browser
• Archive.is - fully loads the webpage, executes JavaScript, rewrites the resulting HTML, and archives the rewritten HTML
• WARCreate - fully loads the webpage, executes JavaScript, and archives the transformed HTML

• CS 725/825 - Information Visualization, Dr. Weigle
• CS 432/532 - Introduction to Web Science, Dr. Nelson

• Justin Brunelledefended his Ph.D. dissertation on February 5, 2016.  Justin already had a full-time position at MITRE, but not coincidentally he had his choice of significant promotions at the conclusion of his Ph.D.
• Yasmin AlNoamanydefendedher Ph.D.dissertation on June 16, 2016.  Yasmin had several post-doctoral opportunities, and eventually decided on a position with Dr. Erik Mitchell, at the UC Berkeley Institute for Data Science (BIDS). 
• Greg Szalkowski completed his M.S. in 2016 as well.  We had hoped to keep him on for a Ph.D., but he's having too much fun traveling the world setting up military communications solutions.

• Shawn Jones passed his breadth exam.
• Alexander Nwala passed his breadth exam.
• Lulwah Alkwai passed her research ability exam.
• Erika Siregar, a new M.S. student, joined us via the Fulbright Scholar Program.

• Shawn's paper "Avoiding spoilers: wiki time travel with Sheldon Cooper", based on his MS Thesis research, was finally published in IJDL.  
• Also in IJDL were expanded versions of three papers from TPDL 2015: Sawood Alam's paper "Web Archive Profiling Through CDX Summarization", and two from Yasmin "Characteristics of social media stories: What makes a good story?" and "Detecting Off-Topic Pages in Web Archives".
• Mat Kelly and Sawood each had a paper at TPDL 2016.  
• Alexander, Sawood, and Mat had three posters at JCDL 2016 (we did not have full paper submissions since Michele Weigle was a PC co-chair).  Alexander's JCDL poster was also released as an expanded tech report. 
• Justin had a paper in D-Lib Magazine about archiving a corporate intranet.
• Shawn's position at LANL was productive, with a paper in PLoS ONE about "Scholarly Context Adrift", a poster at WWW 2016 about linking (or not) to DOIs, and a tech report about extracting HTML from archived web pages, which informed several blog post proposals about formal definitions for getting at raw archived content.  
• Michele had a poster at IEEE Vis 2016 based on a project from her CS 725/825 class.
• I had a tech report with David Rosenthal and Herbert Van de Sompel about various techniques for archiving journals, including Signposting. 

.@WebSciDL on our way to #jcdl2016@machawk1@weiglemc@phonedude_mln@acnwala@ibnesayeed@jcdl2016pic.twitter.com/xI7zeDQeAd

— Michael L. Nelson (@phonedude_mln) June 18, 2016

@elunca@WebSciDL@machawk1@weiglemc@acnwala@ibnesayeed@jcdl2016 yes! Took the ferry to make it longer! ;-) pic.twitter.com/wbJENY334O

— Michael L. Nelson (@phonedude_mln) June 18, 2016

• Justin attended the Federal Cloud Computing Summit in January.
• Sawood, Mat, and Alexander attended the Archives Unleashed Hackathon in March at the University of Toronto.  
• In April I went to both the CNI Spring Meeting in San Antonio and the IIPC General Assembly in Reykjavik.  
• April also saw Shawn attend the WWW Conference in Montreal.
• In May, Erika attended the Fulbright Enrichment Seminar in Pittsburgh. 
• From June through August, Alexander was at Harvard University for a summer fellowship at the Library Innovation Lab; where he worked on the Local Memory Project.
• As mentioned above, in June Mat, Sawood, Alexander, Michele, and I all went to JCDL 2016 (also the Web Archiving & Digital Libraries Workshop, and the JCDL Doctoral Consortium). 
• Right before JCDL 2016 Mat, Sawood, Alexander, Shawn, John Berlin, and Mohamed Aturban attended the second Archives Unleashed Hackathon in Washington DC.  While the rest of the people left for JCDL in New Jersey, Shawn stuck around for the Saving the Web Symposium at the Library of Congress right after the Hackathon. 
• In August I attended the Documenting the Now Advisory Board meeting in St. Louis.
• Mat attended the IIPC Building Better Crawlers Hackathon in London in September.
• In October, Mat, Shawn, John, and I attended the Dodging the Memory Hole meeting at UCLA.
• Sawood went to TPDL 2016 in Hannover in October, and also visited with Dr. Michael Herzog after the conference.  
• Michele attended part of IEEE Vis 2016 in Baltimore in October. 
• And somehow, I don't think anyone traveled in November or December!

.@weiglemc giving program overview #jcdl2016@WebSciDLpic.twitter.com/dJKqdhZ0Z1

— Michael L. Nelson (@phonedude_mln) June 20, 2016

.@WebSciDL celebrates 20 years of #webarchiving& @internetarchive w tacos and @djspooky CDs! #IA20pic.twitter.com/AFb3qUiuzz

— Michael L. Nelson (@phonedude_mln) October 26, 2016

• Mat and Sawood continue to develop IPWB, an implementation of the Wayback Machine and the InterPlanetary Filesystem.  
• Yasmin released her Off-Topic Detection code, as well the story data sets used in her dissertation.
• Zetan Li provided a much needed update for the Carbon Date software. 
• Alexander released the first version of code for the Local Memory Project (from his time @ Harvard).
• Sawood released MemGator, which allows one to setup their own Memento Aggregator.

• "Old Dominion Computer Scientists Awarded $830,000 Mellon Grant"

.@yasmina_anwar returns from @UCBIDS for grad ceremony & avatar advancement @WebSciDL@oducshttps://t.co/Vo2NH3iTpIpic.twitter.com/o02e6nlDc7

— Michael L. Nelson (@phonedude_mln) December 17, 2016

A nanopublication: basic elements from http://nanopub.org

• Trusty URIs can be used to identify and verify resources on the web while in systems like Git version control system, hash values are there to verify "commits" in Git repositories only. The same applies to IPFS where hashes in addresses (e.g., /ipfs/QmZTR5bcpQD7cFgTorqxZDYaew1Wqgfbd2ud9QqGPAkK2V) are used to verify files within the IPFS network only.
• Hashes in trusty URIs can be generated on different kinds of content while in Git or ni-URI, hash values are computed based on the byte level of the content.
• Trusty URIs support self-references (i.e., when trusty URIs are included in the content).

Before changes

After changes

• Kuhn, T., Dumontier, M.: Trusty URIs: Verifiable, immutable, and permanent digital artifacts for linked data. Proceedings of European Semantic Web Conference (ESWC) pp. 395–410 (2014).    
[video, slides shown below]

• Kuhn, Dumontier. "Making Digital Artifacts on the Web Verifiable and Reliable." arXiv preprint arXiv:1507.01697 (2015)
• Kuhn, Tobias. "nanopub-java: A Java Library for Nanopublications." arXiv preprint arXiv:1508.04977 (2015)
• Kurian, S.P., Vishnu S.S.,: Distributed digital artifacts on the semantic web. International Journal of Computer Applications Technology and Research pp. 99–103 (2015)
• IPFS and Multihash
• Klein, M., Van de Sompel, H., Sanderson, R., Shankar, H., Balakireva, L., Zhou, K., Tobin, R.: Scholarly context not found: one in five articles suffers from reference rot. PLoS One 9(12), e115,253 (2014)

A prompt from Twitter indicating no search results

Searching Twitter for a URL by pasting the URL into the search box

Example of a URL that was not found except with the "url:" parameter

Example of a URL that was not found with the "url:" parameter, but found without

Based on these observations, and considering that there was no apparent protocol switching, or URL canonicalization, I scaled the experiment to gain a better insight about this search behavior. I wanted to know the proportion of URLs that are:

1. found exclusively with the "url:" prefix
2. found exclusively without the "url:" prefix
3. found with and without the "url:" prefix (both 1 and 2).

1. Count of URLs found exclusively with the "url:" prefix: 1,519
2. Count of URLs found exclusively without the "url:" prefix: 129
3. Count of URLs found with and without the "url:" prefix (both 1 and 2): 853
4. Count of URLs not found: 1,422

WAIL-Electron

WAIL has been completely revised from a Python application using modern Web technologies into an Electron application. Electron combines a Chromium (Chrome) browser with Node.js allowing for native desktop applications to be created using only HTML, CSS, and JavaScript.

The move to Electron has brought with it many improvements most importantly, of which is the ability to update and package WAIL for the three major operating systems: Linux, MacOS, and Windows. Support for these operating systems is easily achieved by packing utility used (electron-packager) which allows one to produce the binary for a specific system. Also thanks to this move, the directory structure issue mentioned by Mat in his post has been resolved. Electron applications have their own directory structure inside the OS-specific application directory path accessible via their API. Here the packager will place the tools WAIL makes available for use.

Electric Ham

The meat of this revision is adding new functionality to WAIL in addition to the tools already made available through WAIL, namely Heritrix and Wayback. This new functionality comes in two parts. First, WAIL is now collection-centric. The previous revision, WAIL Python, added the WARC files created through WAIL to a single archive. This archive was an ambiguous collection of sorts where users had to create their own means of associating the WARCs to each other. Initially, this beneficial feature allowed users to archive what they saw at any given instance and replay the preserved page immediately. But updates to WAIL could not be justified if they did not build upon the existing functionality; which is why the concept of personal collection-based archiving was introduced.

Collections

WAIL now provides users with the ability for the curation of personalized web archive collections akin to the subscription service Archive-It except on their local machines. By default, WAIL comes with an initial collection and allows for the creation of additional collections.

The Collections screen displays the collections created through WAIL. This view displays the collection name along with some summary information about it.

• Seeds: How many seeds are associated with the collection
• Last Updated: the last time (date and time) the collection was updated
• Size: How large is the collection on the file system

Creation of a collection is as simple as clicking the New Collection button available on the Collections (home) screen of WAIL. After doing so, a dialog will appear from which users can specify the name, title, and description for the collection. Once these fields have been filled in, WAIL will create the collection that users can access from the Collections View.

The Collection View displays the information about each seed contained in the collection

• Seed URL: The URL
• Added: The date time it was added to the Collection
• Last Archived: The last time it was archived through WAIL
• Mementos: The number of Mementos for the seed in the collection

along with a link for viewing the seed in Wayback.

Seeds can be added to a collection from either the live web or from WARC files present on the filesystem. To aid in the process of adding a seed from the live web, WAIL provides the user will the ability to "check" the seed before archiving.

The check provides summary information about the seed that includes the HTTP status code and a report on the embedded resources contained in the page. This lets users choose an archive configuration before starting WAIL's archival process to configure and launch a Heritrix crawl.

To add a seed from the filesystem all the user has to do is drag and drop the (W)ARC file into the corresponding interface for that functionality. WAIL will process the (W)ARC file and display a list of potential seeds discovered.

WAIL can not automatically determine the seed due to the nature of (W)ARC files. Rather WAIL uses heuristics on the contents of the (W)ARC file to determine which entries are valid candidates for the seed URL. From this display, the user chooses the correct one. WAIL will then add the seed to the collection, and it will be available for replay from the Collection View.

Twitter Archiving

The second added functionality is the ability to monitor and archive Twitter content automatically. This was made possible thanks to the scholarship I received for preserving online news. There are two options for the Twitter archival feature implemented in WAIL. The first is monitoring a user’s timeline for tweets which were tweeted after the monitoring has started with the option of selecting only the tweets containing hashtags specified during configuration. The second, a slight variation of the first, will only archive tweets that have specific keywords in the tweet’s body as specified during configuration.

What makes this unique is how WAIL preserves this content. Before this addition, WAIL utilized Heritrix as the primary preservation means. Heritrix executes HTTP GET requests to retrieve the target web page and archives the HTTP response headers and the content returned from the server. The embedded Javascript of the web page is not executed potentially decreasing the fidelity of the capture. This is problematic when archiving Twitter content since the rendering of tweets is done exclusively through client-side Javascript. 

To address this WAIL utilizes the native Chromium browser provided by Electron in conjunction with WARCreate. Modifications were made to WARCreate in order to integrate it with WAIL to eliminate the need for human intervention to decide when to generate the WARC and to work inside of Electron. By integrating WARCreate into WAIL the archival process of Twitter content has been simplified to loading the URL of the tweet into the browser and waiting until the browser indicates that the page has been rendered in its entirety. Then the archival process through WARCreate is initiated. Once the WARC has been generated, it is added to the collection specified by the user.

Putting on Lipstick

As mentioned in Mat's blog post, the UI for WAIL-Python needed an update not only for its maintainability but also for a cohesive user experience across supported platforms. At the time of starting this revision of WAIL, the choices available for the front-end framework as seen on Github were plentiful. It simply boiled down to choosing the one that had the "least" painful setup and deployment process with a learning curve such that any person taking over the project could be brought up to speed with minimal effort.

With this in mind, React was chosen for WAIL's UI library; it is unopinionated about other technologies which may be used alongside it and features a large production tested ecosystem with an active developer community. React is only a view library, which is why WAIL uses Redux and Immutable.js to complete the traditional MVC package. This React, Redux, and Immutable.js stack provide WAIL a consistent user experience across supported platforms and a much more manageable codebase. On the tools side of making WAIL look and perform beautifully, WAIL is now using Ilya Kreymer's pywb. Pywb is used by WAIL for both replay and to aid in the heavy lifting of managing the collections.

WAIL is now available from the project's release page on Github available.  For more information about how to use WAIL be sure to visit the wiki.

- John Berlin

Examples: Archive Now (archivenow) CLI

• A web page can be pushed into one archive
• A web page can be pushed into multiple archives
• A web page can be pushed into all archives  
• Adding new archives
• Removing existing archives

• The CLI  (or A Docker Container)
• A Web Service
• Python code

.@nullhandle presenting the use cases that motivate WASAPI initiative. #webarchivingpic.twitter.com/FixnLqEnnB

— Justin Littman (@justin_littman) February 21, 2017

• Debbie Kempe, NYARC, "NYARC’s Opensearch API Integration"
• Greg Wiedeman, SUNY Albany, "Automating Web Archives Records in ASpace" (see also his blog post from 2016-10-18).
• Stephen Abrams, CDL, "Cobweb"
• Michael Nelson, ODU, "Web Archiving Activities of ODU’s Web Science and Digital Library Research Group" (slides also embedded below)

.@brewster_kahle kicks off day two of the #webArchiving interoperability symposium - in the beautiful sunny @internetarchive great hall. pic.twitter.com/ndlKt9CekF

— Ian Milligan (@ianmilligan1) February 22, 2017

• Tom Cramer, Stanford University, (a talk about collaboration, but I can't find the slides)
• Dallas Pillen, Bentley Historical Library, "ArchivesSpace-Archivematica-DSpace Workflow Integration"
• Nicholas Taylor, Stanford University, (a talk about new developments in LOCKSS, but I can't find the slides)

#WebArchiving in action! pic.twitter.com/7dvsRl95Ut

— Archive-It (@archiveitorg) February 23, 2017

• Justin Littman, GWU, Social Feed Manager (I can't find the slides)
• Ilya Kreymer and Mark Beasley, Rhizome, "Webrecorder Interoperability"
• Ian Milligan (Waterloo) and Nick Ruest (York), "Warcbase: Using Scalable Web Analytics to Analyze Canadian Collections En Masse"

• Matt Weber, Rutgers - Archives Unleashed
• Martin Klein, "Web Archive Interoperability with Memento"

.@mart1nkle1n presenting #memento#webarchivingpic.twitter.com/GUREfN7mJ5

— Michael L. Nelson (@phonedude_mln) February 23, 2017

"No one knows what to do with WARCs, so we got WAIL"#wail@WebSciDL@phonedude_mln  #webarchivingpic.twitter.com/3qi2AKrntI

— Martin Klein (@mart1nkle1n) February 22, 2017

@WebSciDL at @internetarchive after Archives Unleashed 3.0 wrap up. We have a winner of #HackArchivespic.twitter.com/vYLi89yap0

— Sawood Alam (@ibnesayeed) February 25, 2017

Archives Unleashed 3 underway w an overview of Warcbase @ianmilligan1#hackarchives thx @NSF@RutgersCommInfo@internetarchive for support pic.twitter.com/AaJ8Nemvd8

— Matthew Weber (@docmattweber) February 23, 2017

Archives Unleashed day 1: Team EOT-Transition using @eotarchive to try to detect changes in web presence of fed agencies. #hackarchives

— Justin Littman (@justin_littman) February 24, 2017

.@justin_littman gives a lightning intro to the Facebook Graph API and his new tool f(b)arc (https://t.co/7UQtNOiPjZ). #hackarchivespic.twitter.com/uJn17UFGmh

— Ian Milligan (@ianmilligan1) February 24, 2017

Now @walkeroh is showing us how we are collecting social media wrong - users see websites, researchers see scrolling JSON. #hackarchivespic.twitter.com/SWCMT3qq0Z

— Ian Milligan (@ianmilligan1) February 24, 2017

Final presentations at #hackarchives are underway @internetarchive wrapping up archives unleashed 3!! pic.twitter.com/KbV4oFfurh

— Matthew Weber (@docmattweber) February 25, 2017

Next team used Twitter to track "fake news" - how were Trump, Clinton debate quotes shared. How did alternatives appear? #hackarchivespic.twitter.com/NsOtS5eMnt

— Ian Milligan (@ianmilligan1) February 25, 2017

Last up! Analysis of changes in web content across end of term crawls @internetarchive#hackarchivespic.twitter.com/dWrfChnTFF

— Matthew Weber (@docmattweber) February 25, 2017

The final mystery 'team' is @ruebot, who has been exploring Twitter API mysteries. Odd plateaus and peaks in data. #hackarchivespic.twitter.com/Q4dCGBvcTT

— Ian Milligan (@ianmilligan1) February 25, 2017

And the winner is Team Transition @internetarchive#hackarchives !!! That's a wrap folks. pic.twitter.com/bGili0d7WT

— Matthew Weber (@docmattweber) February 25, 2017

And we're proud to announce archives unleashed 4 @britishlibrary jun 11-13. Details at https://t.co/OCbha8LDmk#hackarchives

— Matthew Weber (@docmattweber) February 25, 2017

TL;DR

Background

A State of Replay

www.mendeley.com/sign-in/?routeTo=https%3A%2F%2Fwww.mendeley.com%2Fprofiles%2Fhelen-palmer

Webrcorder

Internet Archives

Comparing The Page On The Live Web To Replay On Archive-It

https://www.mendeley.com/profiles/helen-palmer

https://api.mendeley.com

https://www.mendeley.com/profiles/helen-palmer/co-authors

• GET api.mendeley.com/catalog (x8)
• GET api.mendeley.com/documents (x1)
• GET api.mendeley.com/scopus/article_authors (x8)
• POST api.mendely.com/events/_batch (x1)

http://wayback.archive-it.org/8130/20161215231900/https://www.mendeley.com/profiles/helen-palmerco-authors/

https://www.mendeley.com/profiles/helen-palmer/co-authors

https://www.mendeley.com/profiles/helen-palmerco-authors

http://wayback.archive-it.org/profiles/helen-palmerco-authors

http://wayback.archive-it.org/profiles/helen-palmerco-authors

http://wayback.archive-it.org/profiles/helen-palmerco-authors

http://wayback.archive-it.org/8130/20161215231900/https://www.mendeley.com/profiles/helen-palmerco-authors

http://wayback.archive-it.org/profiles/refreshToken

http://wayback.archive-it.org/8130/20161215231900/https://www.mendeley.com/profiles/refreshToken

Does This Occur In Other Archives

Location, Location, Location

e = t.headers.WB_wombat_self_location;

e && this.settings.followLocation && 201 === t.status

Digging Deeper

api.mendely.com/documents and api.mendely.com/events/_batch

/profiles/helen-palmer/co-authors

anchor_setter herf https://www.mendely.com/profiles/helen-palmer/co-authors

Understanding A Problem By Proxy

https://www.mendeley.com/profiles/helen-palmer

GET request for api.mendeley.com/catalog?=[query string]

Preflighted requests in CORS
In CORS, a preflight request with the OPTIONS method is sent, so that the server can respond whether it is acceptable to send the request with these parameters. The Access-Control-Request-Method header notifies the server as part of a preflight request that when the actual request is sent, it will be sent with a POST request method. The Access-Control-Request-Headers header notifies the server that when the actual request is sent, it will be sent with a X-PINGOTHER and Content-Type custom headers. The server now has an opportunity to determine whether it wishes to accept a request under these circumstances.

application/vnd.mendeley-document.1+json

helen-palmer/co-authors -> helen-palmerco-authors

helen-palmer/co-authors -> helen-palmerco-authors

HTTP Options Request Replay Test

application/vnd.mendeley-document.1+json

application/vnd.mendeley-document.1+json

Conclusions

We performed a study of very large Memento TimeMaps to evaluate the ratio of representations versus redirects obtained when dereferencing each archived capture. Read along below or check out the full report.

Memento represents a set of captures for a URI (e.g., http://google.com) with a TimeMap. Web archives may provide a Memento endpoint that allows users to obtain this list of URIs for the captures, called URI-Ms. Each URI-M represents a single capture (memento), accessible when dereferencing the URI-M (resolving the URI-M to an archived representation of a resource).

Variations in the "original URI" are canonicalized (coalescing https://google.com and http://www.google.com:80/, for instance) with the original URI (URI-R in Memento terminology) also included with a literal "original" relationship value.

<http://ws-dl.blogspot.com/>; rel="original",
<http://web.archive.org/web/timemap/link/http://ws-dl.blogspot.com/>; rel="self"; type="application/link-format"; from="Wed, 29 Sep 2010 00:03:40 GMT"; until="Mon, 20 Mar 2017 19:09:10 GMT",
<http://web.archive.org/web/http://ws-dl.blogspot.com/>; rel="timegate",
<http://web.archive.org/web/20100929000340/http://ws-dl.blogspot.com/>; rel="first memento"; datetime="Wed, 29 Sep 2010 00:03:40 GMT",
<http://web.archive.org/web/20110202180231/http://ws-dl.blogspot.com/>; rel="memento"; datetime="Wed, 02 Feb 2011 18:02:31 GMT",
<http://web.archive.org/web/20110902171049/http://ws-dl.blogspot.com/>; rel="memento"; datetime="Fri, 02 Sep 2011 17:10:49 GMT",
<http://web.archive.org/web/20110902171256/http://ws-dl.blogspot.com/>; rel="memento"; datetime="Fri, 02 Sep 2011 17:12:56 GMT",
...
<http://web.archive.org/web/20151205080546/http://www.ws-dl.blogspot.com/>; rel="memento"; datetime="Sat, 05 Dec 2015 08:05:46 GMT",
<http://web.archive.org/web/20161104143102/http://ws-dl.blogspot.com/>; rel="memento"; datetime="Fri, 04 Nov 2016 14:31:02 GMT",
<http://web.archive.org/web/20161109005749/http://ws-dl.blogspot.com/>; rel="memento"; datetime="Wed, 09 Nov 2016 00:57:49 GMT",
<http://web.archive.org/web/20170119233646/http://ws-dl.blogspot.com/>; rel="memento"; datetime="Thu, 19 Jan 2017 23:36:46 GMT",
<http://web.archive.org/web/20170320190910/http://ws-dl.blogspot.com/>; rel="last memento"; datetime="Mon, 20 Mar 2017 19:09:10 GMT"

For instance, to view the TimeMap for this very blog from Internet Archive, a user may request http://web.archive.org/web/timemap/link/http://ws-dl.blogspot.com/ (Figure 1). Each URI-M (e.g., http://web.archive.org/web/20110902171256/http://ws-dl.blogspot.com/) is listed with a corresponding relationship (rel) and datetime value. Note the www.ws-dl.blogspot.com and ws-dl.blogspot.com subdomain variants are both included in the same TimeMap, an product of the canonicalization procedure. The TimeMap for this URI-R currently contains 60 URI-Ms. Internet Archive's Web interface reports 58 captures -- a subtle yet differing "count". This difference get much more extreme with other URI-Rs.

The quality of each memento (e.g., in terms of completeness of capture of embedded resources) cannot be determined using the TimeMap alone. This fact is inherent in a URI-M needing to be dereferenced and each embedded resource requested upon rending the base URI-M. Comprehensively evaluating the quality over time is something we have already covered (see our TPDL2013, JCDL2014, and IJDL2015 papers/article).

In performing some studies and developing web archiving tools, we required knowing how many captures existed for a particular URI using both a Memento aggregator and the TimeMap from an archive's Memento endpoint. For http://google.com, counting the number of URIs in a TimeMap with a rel value of "memento" produces a count of 695,525 (as of May 2017). The number obtained from Internet Archive's calendar interface and CDX endpoint currently show much smaller count values (e.g., calendar interface currently states 62,339 captures for google.com).

Dereferencing these URI-Ms would take a very long time due to network latency in accessing the archive as well as limits on pipelining (though the latter can be mitigated with distributing the task). We did exactly this for google.com and found that the large majority of the URI-Ms produced a redirect to another URI-M in the TimeMap. This lead us to know that counting mementos in an archive's holdings is not sufficient with this procedure.

Figure 2. Dereferencing URI-Ms may produce a representation, a redirect, or an archived error.

For google.com we found that nearly 85% of the URI-Ms resulted in a redirect when dereferenced. We repeated this procedure for seven other TimeMaps for large web sites (e.g., yahoo.com, instagram.com, wikipedia.org) and found a wide array of trends in this naïve counting method (88.2%, 67.3%, and 44.6% are redirects, respectively). We also repeated this procedure with thirteen academic institutions' URI-Rs to observe if this trend persisted.

We have posted an extensive report of our findings as a tech report available on arXiv (linked below).

— Mat (@machawk1)

Mat Kelly, Lulwah M. Alkwai, Michael L. Nelson, Michele C. Weigle, and Herbert Van de Sompel. "Impact of URI Canonicalization on Memento Count," Technical Report arXiv:1703.03302, 2017.

This is where students go most often. Interesting how many are free resources.  None are for-profits (well, maybe course readings). #cni17spic.twitter.com/xHJGgA3gq2

— Roger C. Schonfeld (@rschon) April 3, 2017

Great "direct from the swamp" public policy update from @ARLpolicy& ALA public policy #cni17spic.twitter.com/vQfN8FiEJw

— Dave Hansen (@DigLibCopyright) April 3, 2017

Your calls are paying off! So far we have 108 signers for LSTA and 136 for IAL. There's still time to act https://t.co/lF5OI2QpZ0#SaveIMLSpic.twitter.com/abLKrUV2ts

— Amer. Library Assn. (@ALALibrary) April 3, 2017

Interesting study and findings on rescuing scholarly orphans by @hvdsomp@mart1nkle1n@phonedude_mln#cni17spic.twitter.com/YFdejE0KZE

— Yasmina Anwar (@yasmina_anwar) April 3, 2017

@amy_brand Cliff Lynch thanks Amy for great presentation.  I agree, very informative! #cni17spic.twitter.com/GsiesTncAp

— Robert Cartolano (@rob_cartolano) April 4, 2017

@amy_brand Strategic roadmap #cni17spic.twitter.com/Q89bAS3BR2

— Robert Cartolano (@rob_cartolano) April 4, 2017

Brand: Why MIT Press would build its own  platforms/subscriptions? #cni17spic.twitter.com/siJfRoGcYE

— Mary Ellen Davis (@med744) April 4, 2017

Institutional & Web Archiving perspectives to rescue scholarly orphans! #cni17spic.twitter.com/Xt59pkNUu8

— Yasmina Anwar (@yasmina_anwar) April 3, 2017

ORCID @ORCID_Org  is dominated by life sciences! #cni17s@mart1nkle1npic.twitter.com/sJhYxJ80kO

— Yasmina Anwar (@yasmina_anwar) April 3, 2017

.@phonedude_mln on legal challenges re capturing scholarly artifacts - it's a mess! w/ @hvdsomp and myself #cni17spic.twitter.com/YEg3xnMcDJ

— Martin Klein (@mart1nkle1n) April 3, 2017

Day 1

Gary Wolf kicking off #pda2017. pic.twitter.com/X3pUOEddNs

— John Berlin (@johnaberlin) March 29, 2017

@agaricus Gary Wolf talking abt the quantified self. groups meet to discuss #lifelogging& self tracking https://t.co/MnNPDJcYiA… #pda2017

— Melody Condron (@MTbeekeeper) March 29, 2017

Dorothy Waugh & Elizabeth Russey Roke,
Second-Generation Digital Archives: What We Learned from the Salman Rushdie Project #pda2017pic.twitter.com/d2mfv6OQUN

— John Berlin (@johnaberlin) March 29, 2017

#pda2017 Emory opted to provide two tiers of access in reading room for Rushdie CPUs. 1) PDFs of content 2) emulation of CPU environment

— robin margolis (@poeticdoxa) March 29, 2017

Jessica Moran Composing an Archive: the personal digital archives of contemporary composers in New Zealand #pda2017pic.twitter.com/kbMnaWztFY

— John Berlin (@johnaberlin) March 29, 2017

#pda2017@jessicammoran shares on composer personal archive featuring musician stand-bys of floppy disks and drafts of composition on Logic!

— robin margolis (@poeticdoxa) March 29, 2017

Rachel Foss Learning from users of personal digital archives at the British Library #pda2017pic.twitter.com/JgvyFBQKgk

— John Berlin (@johnaberlin) March 29, 2017

#pda2017 Rachel Foss shares choice to allow user access to unreadable files, offering teaching moments around digital preservation

— robin margolis (@poeticdoxa) March 29, 2017

Charles Ransom moderating Teaching PDA #pda2017pic.twitter.com/jcbQCwsOSO

— John Berlin (@johnaberlin) March 29, 2017

Model for Journal Archive Management #JAM#pda2017#Journalspic.twitter.com/4acegUqGqc

— Yasmina Anwar (@yasmina_anwar) March 29, 2017

Journalism Archive Management (JAM) program @Mizzou teaches communications students to create, label, store, find +reuse digi files #pda2017

— Robin M. Katz (@robinmkatz) March 29, 2017

Carly Dearborn: An archivist in the lab with a codebook #pda2017pic.twitter.com/dfHFauB8y4

— John Berlin (@johnaberlin) March 29, 2017

W her campus population, @carlydearborn found data producers just wanted step-by-step instructions, not to think critically abt it #pda2017

— Robin M. Katz (@robinmkatz) March 29, 2017

Nicholas Taylor moderating Emergent Technologies & PDA 1#pda2017pic.twitter.com/BjsIcP1je9

— John Berlin (@johnaberlin) March 29, 2017

Maria Praetzellis & Jefferson Bailey
Cogifo Ergo Sum: GifCities & Personal Archives on the Web #pda2017pic.twitter.com/Sufvb56EHE

— John Berlin (@johnaberlin) March 29, 2017

@jefferson_bail& maria praetzellis present on geocities and personal archives on the web #pda2017pic.twitter.com/ZswptEB5nV

— ePADD (@e_padd) March 29, 2017

Melody Condron Comparison of Aggregate Tools for Archiving Social Media #pda2017pic.twitter.com/1mBxFtdfrS

— John Berlin (@johnaberlin) March 29, 2017

"You can make Twitter to turn into yr coffeemaker", @MTbeekeeper! Gr8 tools for aggregating social media #pda2017pic.twitter.com/i2zGcprOlG

— Yasmina Anwar (@yasmina_anwar) March 29, 2017

Adam Lefloic Lebel, Video games collectors and archivists: how might private archives influence archival practices #pda2017pic.twitter.com/5BXVGz0sNm

— John Berlin (@johnaberlin) March 29, 2017

Collectors help with preserving games - Research from the University of Montreal @melswal#pda2017pic.twitter.com/iVO6MCxRBy

— Sarah Slade (@sladiladi) March 29, 2017

.@johnaberlin from @WebSciDL@oducs is demoing #WAIL@StanfordLibs#pda2017pic.twitter.com/htNqLae4Ib

— Yasmina Anwar (@yasmina_anwar) March 29, 2017

Day 2

• 5,992 US Newspapers
• 1,061 US TV stations, and
• 2,539 US Radio stations

{
"city": "Cambridge", 
"city-latitude": 42.379146, 
"city-longitude": -71.12803, 
"collection": [
   {
"city-county-lat": 42.377, 
"city-county-long": -71.1167, 
"city-county-name": "Harvard", 
"country": "USA", 
"facebook": "http://www.facebook.com/TheHarvardCrimson", 
"media-class": "newspaper", 
"media-subclass": "college", 
"miles": 0.6, 
"name": "Harvard Crimson", 
"open-search": [], 
"rss": [], 
"state": "MA", 
"twitter": "http://www.twitter.com/thecrimson", 
"video": "https://www.youtube.com/user/TheHarvardCrimson/videos", 
"website": "http://www.thecrimson.com/"
   }
  ], 
"country": "USA", 
"self": "http://www.localmemory.org/api/countries/USA/02138/10/?off=tv%20radio%20", 
"state": "MA", 
"timestamp": "2017-04-17T18:56:10Z"
 }

The Memento Protocol provides a Memento-Datetime header indicating at what datetime a memento was captured by a given web archive. In most cases, this metadata sufficiently informs the user of when the given web resource existed. Even though it has been established in US courts that web archives can be used to legally establish that a given web resource existed at a given time, there is still potential to doubt this timestamp because the same web archive that provides the memento also provides its Memento-Datetime. Though not a replacement for Memento-Datetime, trusted timestamping is the process that provides certainty of timestamps for content and can be used to provide additional data to alleviate this doubt.

Brief Cryptography Background

Trusted timestamping systems use some concepts from cryptography for confidentiality and integrity. I will provide a brief overview of these concepts here.

Throughout this document I will use the verb hashing to refer to the use of a one-way collision-resistant hash function. Users supply content, such as a document, as input and the hash function provides a digest as output.  Hash functions are one-way, meaning that no one can take that digest and reconstitute the document. Hash functions are collision-resistant, meaning that there is a very low probability that another input will produce the same digest, referred to as a collision. As shown in the figure below, small changes to the input of a hash function produce completely different digests. Thus, hash digests provide a way to identify content without revealing it. The output of the hash function is also referred to as a hash.

This diagram shows the digests produced with the same cryptographic hash function over several inputs. A small change in the input produces a completely different hash digest as output. Source: Wikipedia

The timestamping solutions in this post use the SHA-256 and RIPEMD-160 hash functions. SHA-256 is a version of the SHA-2 algorithm with 256 bit keys. Its predecessor, SHA-1, has been under scrutiny for some time. In 2005, cryptographers showed mathematically that SHA-1 was not collision-free, prompting many to start moving to SHA-2. In February of 2017, Google researchers were able to create a collision with SHA-1, showing that SHA-1 is no longer reliably trustworthy. Because collision attacks are theoretically possible, though technically infeasible, for SHA-2, SHA-3 has been developed as a future replacement. I mention this to show how the world of hash functions is dynamic, resulting in continued research of better functions. For this post, however, it is most important to just understand the purpose of hash functions.

In addition to hash functions, this post discusses solutions that utilize pubic-key cryptography, consisting of private keys and public keys. Users typically generate a private key using random information generated by their computer and an algorithm such as 3DES. Users then use this private key with an algorithm such as RSA or ECC to generate a public key. Users are expected to secure their private key, but share the public key.

A diagram showing an example of encryption using public and private keys. Source: Wikipedia

Users use public keys to encrypt content and private keys to decrypt it. In the figure above, everyone has access to Alice's pubic key. Bob encrypts a message using Alice's public key, but only Alice can decrypt it because she is the only one with access to her private key.

This process can be used in reverse to digitally sign content. The private key can be used to encrypt content and the public key can be used to decrypt it. This digital signature allows anyone with access to the public key to verify that the content was encrypted by the owner of the private key because only the owner of the private key should have access to the private key.

Certificates are documents containing a public key and a digital signature. A user typically requests a certificate on behalf of themselves or a server. A trusted certificate authority verifies the user's identity and issues the certificate with a digital signature. Other users can verify the identity of the owner of the certificate by verifying the digital signature of the certificate with the certificate authority. Certificates expire after some time and must be renewed. If a user's private key is compromised, then the certificate authority can revoke the associated certificate.

A nonce is a single-use value that is added to data prior to encryption or hashing. Systems often insert it to ensure that transmitted encrypted data can not be reused by an attacker in the future. In this article nonces are used with hashing as part of Bitcoin's proof-of-work function, to be explained later.

Finally, there is the related concept of binary-to-text encoding. Encoding allows a system to convert data to printable text. Unlike hash digests, encoded text can be converted back into its original input data. Cryptographic systems typically use encoding to create human-readable versions of public/private keys and hash digests. Base64 is a popular encoding scheme used on the Internet. Bitcoin also uses the lesser known Base58 scheme.

Brief Bitcoin Background

Bitcoin is a cryptocurrency. It is not issued by an institution or backed by quantities of physical objects, like gold or silver. It is software that was released with an open source license to the world by an anonymous individual using the pseudonym Satoshi Nakamoto. Using a complex peer-to-peer network protocol it ensures that funds (Bitcoins) are securely transferred from one account to another.

Bitcoin accounts are identified by addresses. Addresses are used to indicate where Bitcoins should be sent (paid). The end user’s Bitcoin software uses public and private keys to generate an address. Users often have many addresses to ensure their privacy. Users have special purpose software, called Wallets, that generates and keeps track of addresses. There is no central authority to issue addresses, meaning that addresses must be generated individually by all participants.

Wallets generate Bitcoin addresses using the following process:

1. Generate an ECC public-private key pair
2. Perform SHA-256 hash of public key
3. Perform RIPEMD-160 hash of that result
4. Add version byte (0x00) to the front of that result
5. Perform a SHA-256 hash of that result, twice
6. Append the first 4 bytes of that result to the value from #4
7. Convert that result into Base58, which eliminates confusing characters 0 and O as well as 1 and l

The last step uses Base58 so that users can write the address on a piece of paper or speak it aloud over the phone. The ECC algorithms are used by Bitcoin to make the public-private key pair "somewhat resistant" to quantum computers. SHA-256 is used twice in step 5 to reduce the chance of success for any as yet unknown attacks against the SHA-2 hash function. Because all Bitcoin users generate addresses themselves, without a central addressing authority, this long process exists to reduce the probability of a collision between addresses to 0.01%. Even so, for improved security, the community suggests generating new addresses for each transaction. Note that only public-private keys and hashing are involved. There are no certificates to revoke or expire.

Transactions contain the following types of entries:

• Transaction inputs contain a list of addresses and amount of Bitcoins to transfer from those addresses. Also included is a digital signature corresponding to each address. This digital signature is used by the Bitcoin software to verify that the transaction is legitimate and thus these Bitcoins can be spent. There is also a user-generated script used to specify how to access the bitcoins, but the workings of these scripts are outside the scope of this post.
• Transaction outputs contain a list of addresses and amount of Bitcoins to transfer to those addresses. As with transaction inputs, a user-generated script is included to specify how to spend the bitcoins, but I will not go into further detail here.
• Another field exists to enter the amount of transaction fees paid to the miners for processing the transaction.

The Bitcoin system broadcasts new transactions to all nodes. Miners select transactions and group them into blocks. A block contains the transactions, a timestamp, a nonce, and a hash of the previous block.

Within each block, Bitcoin stores transactions in a Merkle tree, an example diagram of which is shown below. Transactions reside in the leaves of the tree. Each non-leaf node contains a hash of its children. This data structure is used to prevent corrupt or illicit transactions from being shared, and thus included in the block chain.

A diagram showing an example of a Merkle tree. Each non-leaf node contains a hash of its children. For Bitcoin, transactions reside in the leaves. Source: Wikipedia

A conceptual diagram shows the Bitcoin blockchain. Each block contains: a hash of the previous block, a timestamp, a nonce, and the root of a tree of transactions. Source: Wikipedia

Miners only see Bitcoin addresses and amounts in each transaction, providing some privacy to those submitting transactions. To add a block to the blockchain, miners must solve a proof-of-work function. Once a block has been assembled by a miner, the Bitcoin software generates a nonce, adds it to the content of the block, and then hashes the contents of the block with the nonce using SHA-256, twice, to produce a hash. The system does not share the nonce with the miners. To add their block to the Bitcoin blockchain, the miner must guess nonces, combine them with the block content, and hash this content until they produce the correct hash digest value. This proof-of-work function is designed to be fast for the system to verify, but time-consuming for the miners to execute. The length of the nonce is increased every 14 days to maintain the level of difficulty in solving the proof-of-work function. This value was chosen to ensure that that miners continue to take 10 minutes to process each block. For each block completed, miners are rewarded any user-provided transaction fees included in the transactions as well as newly minted Bitcoins -- a block reward. The block reward is currently set at 12.5 bitcoins, worth $15,939 as of March 2, 2017. Miners run software and dedicated hardware around the globe to solve the proof-of-work function. Currently the local cost of electricity is the limiting factor in profiting from mining bitcoins.

To alter previous transactions, an attacker would need to select the block of the transaction they wished to alter. They would then need to insert their illicit transaction into the block and create a new block. After this they would then need to solve the block containing that transaction and all subsequent blocks faster than more than 50% of all the other miners, thus it is considered to be extremely hard to alter the blockchain.

Bitcoins do not really exist, even on an individual's hard drive. The blockchain contains the record of every bitcoin spent and indicates the current balance at each Bitcoin address. Full Bitcoin nodes have a copy of the blockchain, currently at 105GB, which can create problems for users running full nodes. Satoshi Nakamoto recommended periodically pruning the blockchain of old transactions, but so far this has not been done.

Technology exists to create blockchains outside of Bitcoin, but Bitcoin provides incentives for participation, in terms of monetary rewards. Any system attempting to use a blockchain outside of Bitcoin would need to produce similar incentives for participants. The participation by the miners also secures the blockchain by preventing malicious users from spamming it with transactions.

How accurate are the timestamps in the blockchain? According to the Bitcoin wiki:

A timestamp is accepted as valid if it is greater than the median timestamp of previous 11 blocks, and less than the network-adjusted time + 2 hours. "Network-adjusted time" is the median of the timestamps returned by all nodes connected to you. As a result, block timestamps are not exactly accurate, and they do not even need to be in order. Block times are accurate only to within an hour or two.

Bitcoins come in several denominations. The largest is the Bitcoin. The smallest is the satoshi. One satoshi equals 0.00000001 (1 x 10^-8) Bitcoins.

Trusted Timestamping

Trusted Timestamping with RFC 3161 and ANSI X9.95

1. The requestor creates a hash of the content.
2. The requestor submits this hash to the TSA.
3. The TSA ensures that its clock is synchronized with an authoritative time source.
4. The TSA ensures that the hash is the correct length, but, to ensure privacy, does not examine the hash in any other way.
5. The TSA generates a TST containing the hash of the document, the timestamp, and a digital signature of these two pieces of data. The digital signature is signed with a private key whose sole purpose is timestamping. RFC 3161 requires that the requestor not be identified in the TST. The TST may also include additional metadata, such as the security policy used.
6. The TST is sent back to the requestor, who should then store it along with the original document for future verification.

Simplified diagram showing the process of using a Time Stamp Authority (TSA) to issue and verify timestamps. Source: Wikipedia

1. The verifier verifies the digital signature of the TST against the TSA’s certificate. If this is correct, then they know that the TST was issued by the TSA.
2. The verifier then checks that the hash in the TST matches the hash of the document. If they match, then the verifier knows that the document hash was used to generate that TST.
3. The timestamp contained in the TST and the hash were used in the generation of the digital signature, hence the TSA observed the document at the given time.

Trusted Timestamping with OriginStamp

They created the OriginStamp system for timestamping user-submitted documents with the Bitcoin blockchain. They chose Bitcoin because it is the most widely used cryptocurrency and thus is perceived to last for a long time. This longevity is a requirement for verification of timestamps in the future.

OriginStamp Process to convert a document content into a Bitcoin address for use in a Bitcoin transaction that can be later verified against the blockchain.

The figure above displays the OriginStamp process for creating a Bitcoin address from a document:

1. A user submits a document to the system which is then hashed, or just submits the hash of a document.
2. The submitted document hash is placed into a list of hashes -- seed text -- from other submissions during that day.
3. Once per day, this seed text is itself hashed using SHA-256 to produce an aggregate hash.
4. This aggregate hash is used as the Bitcoin private key which is used to generate a public key.
5. That public key is used to generate a Bitcoin address which can be used in a timestamped Bitcoin transaction of 1 satoshi.

OriginStamp could submit each document hash to the blockchain as an individual transaction, but the hashes are aggregated together to keep operating costs low. Because fees are taken out of every Bitcoin transaction, each transaction costs $ 0.03, allowing Gipp and his team to offer this low cost service for free. They estimate that the system costs $10/year to operate.

Their paper was published in March of 2015. According to coindesk.com, for the March 2015 time period 1 Bitcoin was equal to $268.32. As of March of 2017, 1 Bitcoin is now equal to $960.36. The average transaction fee now sits at approximately 45,200 satoshis, resulting in a transaction fee of $0.43, as of March 26, 2017.

A screenshot of the memento that I timestamp throughout this section.

OriginStamp allows one to submit documents for timestamping using the Bitcoin blockchain. In this case, I submitted the HTML content of the memento shown in the figure above.

OriginStamp responds to the submission by indicating that it will submit a group of hashes to the Bitcoin blockchain in a few hours.

With the OriginStamp service, the requestor acquires a timestamp using the following process:

1. Submit the document -- or just its hash -- to the OriginStamp website as seen in the screenshots above. If a document is submitted, its hash is calculated and the document is not retained.
2. OriginStamp sends an email once the system has submitted the concatenated hashes to the Bitcoin blockchain. This email will contain information about the seed text used and this seed  must be used for verification.
3. In addition, the @originstamp Twitter account will tweet that the given hash was submitted to the blockchain.

A screenshot showing how OriginStamp displays verification information for a given hash. In this case, the document hash is da5328049647343c31e0e62d3886d6a21edb28406ede08a845adeb96d5e8bf50 and it was submitted to the blockchain on 4/10/2017 10:18:29 AM GMT-0600 (MDT) as part of seed text whose hash, and hence private key is c634bcafba86df8313332abc0ae854eea9083b279cdd4d9cde1d516ee6fb70d9.

Because the blockchain is expected to last for the forseeable future and is tamper-proof, it can be used at any time to verify the timestamp. There are two methods available: with the OriginStamp service, or directly against the Bitcoin blockchain using the seed text.

To do so with the OriginStamp service, the verifier can follow this process:

1. Using the OriginStamp web site, the verifier can submit the hash of the original document and will receive a response as shown in the screenshot above. The response contains the timestamp under the heading "Submitted to Bitcoin network".
2. If the verifier wishes to find the timestamp in the blockchain, they can expand the "Show transaction details" section of this page, shown below. This section reveals a button allowing one to download the list of hashes (seed text) used in the transaction, the private and public keys used in the transaction, the recipient Bitcoin address, and a link to blockchain.info also allowing verification of the transaction at a specific time.
3. Using the link "Verify the generated secret on http://blockchain.info", they can see the details of the transaction and verify the timestamp, shown in the figure below.

A screenshot showing that more information is available once the user clicks "show transaction details". The recipient Bitcoin address is outlined in red. From this screen, the user can download the seed text containing the list of hashes submitted to the Bitcoin blockchain. A SHA-256 hash of this seed text is the Bitcoin private key. From this private key, a user can generate the public key and eventually the Bitcoin address for verification. In this case the generated Bitcoin address is 1EcnftDQwawHQWhE67zxEHSLUEoXKZbasy.

A screenshot of the blockchain.info web site showing the details of a Bitcoin transaction, complete with its timestamp. The Bitcoin address and timestamp have been enlarged and outlined in red. For Bitcoin address 1EcnftDQwawHQWhE67zxEHSLUEoXKZbasy, 1 Satoshi was transferred on 2017-02-28 00:01:15, thus that transaction date is the timestamp for the corresponding document.

To verify that a document was timestamped by directly checking the Bitcoin blockchain, one only needs:

1. Generate the hash of the document.
2. Verify that this hash is in the seed text. This seed text should have been saved as a result of the email or tweet from OriginStamp.
3. Hash the seed text with SHA256 to produce the Bitcoin private key.
4. Generate the Bitcoin address using a tool such as bitaddress.org. The figure below shows the use of bitaddress.org to generate a Bitcoin address using a private key.
5. Search the Bitcoin blockchain for this address, using a service such as blockchain.info. The block timestamp is the timestamp of the document.

A screenshot of bitaddress.org's "Wallet Details" tab with the Bitcoin address enlarged and outlined in red. One can insert a Bitcoin private key and it will generate the pubic key and associated Bitcoin address. This example uses the private key of c634bcafba86df8313332abc0ae854eea9083b279cdd4d9cde1d516ee6fb70d9 shown in previous screenshots which corresponds to a Bitcoin address of 1EcnftDQwawHQWhE67zxEHSLUEoXKZbasy, also shown in previous figures.

OriginStamp also supplies an API that developers can use to submit documents for timestamping as well as verify timestamps and download the seed text for a given document hash.

Others are using blockchain technology for building a decentralized Internet and creating applications. Some Bitcoin users are not happy with using the Bitcoin blockchain for non-currency related transactions for fear that the miners' time will be taken up with transactions that are not related to commerce. Other Bitcoin users are quite happy with using it for timestamping. IBM is creating their own blockchain technologies for use in applications.

Comparison of Timestamping Solutions

In the table below I provide a summary comparison between the TSA, Origin Stamp, and submitting directly to the blockchain without OriginStamp.

	TSA	OriginStamp	Directly To Blockchain
Financial Cost per Timestamp	Dependent on Service and subscription, ranges from $3 to $0.024	Dependent on size of seed text, but less than Bitcoin transaction fee	Bitcoin transaction fee, optimally $0.56
Accuracy of Timestamp	Within seconds of time of request, but dependent on number of requests in queue if linked timestamping used	Within 1 Day + 2 hours	Within 2 hours
Items Needed For Verification	Original Document TST Certificate of server to verify signature	Original Document The seed text of hashes submitted at the same time	Original Document
Tools Needed For Verification	Certificate verification tools	Software to generate Bitcoin Address Software to search blockchain
Timestamp Storage	In TST saved by requestor	Blockchain
Privacy	Only hash of document is submitted, but TSA knows of the requestor's IP address	Miners only see the Bitcoin address, not who submitted the document or even its hash
Targets of Compromise	TSA time server TSA certificate private key	Blockchain
Requirement for Compromise	Server is configured insecurely Server has open software vulnerabilities	>50% of Bitcoin miners colluding
Dependency of Longevity	Life of Organization Offering Timestamping Service	Continued Interest In Preserving the Blockchain

In the first row, we compare the cost of timestamps from each service. At the TSA service run by Digistamp, an archivist can obtain a cost of $0.024 per timestamp for 600,000 timestamps, but would need to commit to an 1 year fee of $14,400. They would also need to acquire all 600,000 timestamps within a year or lose them. If they pay $10, they are only allocated 30 timestamps and need to use them within 3 months, resulting in a cost of $3 per timestamp. Tecxoft's pricing is similar. OriginStamp attempts to keep costs down by bundling many hashes into a seed text file, but is still at the mercy of Bitcoin's transaction fees. The price of Bitcoin is currently very volatile. The transaction fee mentioned in Gipp's work from 2015 was $0.03. Miners tend to process a block faster if it has a higher transaction fee. The optimal fee has gone up due to Bitcoin's rise in value and an increase in the number of transactions waiting to be processed. The lowest price for the least delay is currently 200 satoshis per byte and the median transaction size is 226 bytes for a total cost of 45,200 satoshis.  This was equivalent to $0.43 when I started writing this blog post and is now $0.56.

In the second row, we compare timestamp accuracy. The TSA should be able to issue a timestamp to the requestor within seconds of the request. This can be delayed by a long queue if the TSA uses linked timestamping because every request must be satisfied in order. OriginStamp, however, tries to keep costs down by submitting its seed list to the blockchain at once-a-day intervals, according to the paper. On top of this, the timestamp in the blockchain is accurate to within two hours of submission of the Bitcoin transaction. This means that an OriginStamp timestamp may be as much as 24 hours + 2 hours = 26 hours off from the time of submission of the document hash. In practice, I do not know the schedule used by OriginStamp, as I submitted a document on February 28, 2017 and it was not submitted to the Bitcoin network until March 4, 2017. Then again, a document submitted on March 19, 2017 was submitted to the Bitcoin network by OriginStamp almost 18 hours later.

If the cost is deemed necessary, this lack of precision can be alleviated by not using OriginStamp but submitting to the blockchain directly. One could generate a Bitcoin address from a single document hash and then submit it to the blockchain immediately. The timestamp precision would still be within 2 hours of transaction submission.

For the TSA, timestamps are stored in the TST, which must be saved by the requestor for future verification. In contrast, OriginStamp saves timestamps in the Blockchain. OriginStamp users still need to save the seed list, so both solutions require the requestor to retain something along with the original document for future verification.

All solutions offer privacy through the use of document hashes. The Bitcoin miners receiving OriginStamp transactions only see the Bitcoin address generated from the hash of the seed list and do not even know it came from OriginStamp, hiding the original document submission in additional layers. The TSA, on the other hand, is aware of the requestor's IP address and potentially other identifying information.

To verify the timestamp, TSA users must have access to the original document, the TST, and the certificate of the TSA to verify the digital signature. OriginStamp only requires the original document and the seed list of hashes submitted to the blockchain. This means that OriginStamp requires slightly fewer items to be retained.

If using the blockchain directly, without OriginStamp, a single document hash could be used as the private key. There would be no seed list in this case. For verification, one would merely need the original document, which would be retained anyway.

To compromise the timestamps, the TSA time server must be attacked. This can be done by taking advantage of software vulnerabilities or insecure configurations. TSAs are usually audited to prevent insecure configurations, but vulnerabilities are frequently discovered in software. OriginStamp, on the other hand, requires that the blockchain be attacked directly, which is only possible if more than 50% of Bitcoin miners collude to manipulate blocks.

Finally, each service has different vulnerabilities when it comes to longevity. Mementos belong to web archives, and as such, are intended to exist far, far longer than 20 years. This makes longevity a key concern in any decision of a timestamping solution. The average lifespan of a company is now less than 20 years and expected to decrease. The certificates for verifying a timestamp running a TSA may last for little more than 3 years. This means that the verifier will need someone to have saved the TSA's certificate prior to verification of the timestamp. If the organization with the document and the TST is also the same organization providing the TSA's certificate, then there is cause for doubt in its validity because that organization can potentially forge any or all of these verification components.

The Bitcoin blockchain, on the other hand, is not tied to any single organization and is expected to last as long as there is interest in investing in the cryptocurrency. In addition, there are many copies of the blockchain available in the world. If Bitcoin goes away, there is still an interest in maintaining the blockchain for verification of transactions, and thus retaining copies of the blockchain by many parties. If someone wishes to forge a timestamp, they would need to construct their own illicit blockchain. Even if they went that far, a copy of their blockchain can be compared to other existing copies to evaluate its validity. Thus, even if the blockchain is no longer updated, it is still an independent source of information that can be used for future verification. If the blockchain is ever pruned, then prior copies will still need to be archived somewhere for verification of prior transactions. The combined interests of all of these parties support the concept of the Bitcoin blockchain lasting longer than a single server certificate or company.

So, with trusted timestamping available, what options do we have to make it easy to verify memento timestamps?

Trusted Timestamping of Mementos

It is worth noting that, due to delays between timestamp request and response in each of these solutions, trusted timestamping is not a replacement for Memento-Datetime. The Memento-Datetime header provides a timestamp of when the web archive captured a given resource. Trusted timestamping, on the other hand, can provide an additional dimension of certainty that a resource existed at a given datetime. Just as Memento-Datetime applies to a resource at a specific URI-M, so would a trusted timestamp.

A crawler can capture a memento as part of its normal operations, compute the hash of its content, and then submit this hash for timestamping to one of these services. The original memento content encountered during the crawl, the raw memento, must be preserved by the archive indefinitely. The memento can include a link relation in the response headers, such as the unregistered trusted-timestamp-info relation shown in the example headers below, indicating where one can find additional information to verify the timestamp.

HTTP/1.1 200 OK Server: Tengine/2.1.0 Date: Thu, 21 Jul 2016 17:34:15 GMT Content-Type: text/html;charset=utf-8 Content-Length: 109672 Connection: keep-alive Memento-Datetime: Thu, 21 Jul 2016 15:25:44 GMTLink: <http://www.cnn.com/>; rel="original", <http://a.example.org/web/timemap/link/http://www.cnn.com/>; rel="timemap"; type="application/link-format", <http://a.example.org/web/http://www.cnn.com/>; rel="timegate", <http://a.example.org/web/20160721152544/http://www.cnn.com/>; rel="last memento"; datetime="Thu, 21 Jul 2016 15:25:44 GMT", <http://a.example.org/web/20160120080735/http://www.cnn.com/>; rel="first memento"; datetime="Wed, 20 Jan 2016 08:07:35 GMT", <http://a.example.org/web/20160721143259/http://www.cnn.com/>; rel="prev memento"; datetime="Thu, 21 Jul 2016 14:32:59 GMT", <http://a.example.org/timestamping/20160722191106/http://www.cnn.com/>; rel="trusted-timestamp-info"

The URI linked by the timestamp-info relation could be a JSON-formatted resource providing information for verification. For example, if OriginStamp is used for timestamping, then the resource might look like this:

{ "timestamping-method": "OriginStamp", "seed-text": "http://a.example.org/timestamping/20160722191106/seedtext.txt",
"hash-algorithm": "SHA-256" }

In this case, a verifier already knows the URI-M of the memento. They only need to locate the raw memento, calculate its hash, and use the seed text as described above to generate the Bitcoin address and find the timestamp in the Bitcoin blockchain.

Or, if an RFC 3161 solution is used, the timestamp-info resource could look like this:

{ "timestamping-method": "RFC 3161", "timestamp-token": "http://a.example.org/timestamping/tst/20160721174431/http://www.cnn.com", "tsa-certificate": "http://a.example.org/timestamping/tsacert/cert.pem",
"hash-algorithm": "SHA-256" }

In this case, a verifier can locate the raw memento, calculate its hash, and use verify it using the timestamp token (TST) and the TSA certificate as described above for RFC 3161.

If it is known that the crawler creates the hash of the raw memento and uses it as a private key for generating a Bitcoin address, thus submitting it directly to the blockchain for timestamping, then no additional headers would be needed. Verifiers only need the content of the raw memento to generate the hash. In addition, perhaps a separate timestamping service could exist for mementos, using the URI-M (e.g., https://timestamping-service.example.org/{URI-M}).

If one specific timestamping scheme is used, then perhaps specific link relations can be created to convey the resources from each of these fields.

Of course, this assumes that we only want to timestamp raw mementos. Conceivably, one might wish to timestamp a screenshot of a web page, or its WARC. We will need to perform additional analysis of the other potential use cases needed.

Summary

In this post, I have discussed different timestamping options. These options have different levels of complexity and security. All of them support privacy by a permitting the submission of a document hash to a timestamping service. OriginStamp attempts to address some of the concerns of the existing ANSI X9.95/RFC 3161 standard by using the timestamping features of the Bitcoin blockchain.

Of these options, the Bitcoin blockchain offers a decentralized, secure solution that supports privacy and does not depend on a central server that can fail or be compromised. Because copies of the blockchain are distributed to all full Bitcoin clients, it remains present for verification in the future. Bitcoin has been around for 8 years and continues to increase in value. Because all participants have incentives to keep the blockchain distributed and up to date, it is expected to outlast most companies, who have a median age of 20 years. In addition, if Bitcoin is no longer used, copies of the blockchain will still need to be maintained indefinitely for verification. It does, however, suffer from issues with timestamp accuracy inherent in the Bitcoin protocol. These can be alleviated by submitting a document hash directly against the blockchain.

Companies offering trusted timestamping using TSAs, on the other hand, may not have the longevity and require subscription fees for a limited number of timestamps. Though Bitcoin is currently volatile, it has stabilized before, and the subscription fees from these companies are still more expensive on average than the Bitcoin transaction fee.

Even though timestamping options exist, use cases must be identified for the verification of such timestamps in the future. These use cases will inform requestors of which content to be timestamped and will also affect which timestamping solution is selected. It would also be beneficial for verifiers to have links to additional resources for verification.

Trusted timestamping of mementos is possible, but will require some additional decisions and technology to become a reality.

Additional References Used For This Post:

• Bela Gipp, Norman Meuschke, and André Gernandt. 2015. Trusted Timestamping using the Crypto Currency Bitcoin. In iConference 2015. Newport Beach, CA: iSchools. http://hdl.handle.net/2142/73770
• Pedro Franco. 2015. Understanding Bitcoin: Cryptography, Engineering and Economics, Chichester, UK: John Wiley & Sons Ltd. http://www.wiley.com/WileyCDA/WileyTitle/productCd-1119019168.html
• Satoshi Nakamoto. 2008. Bitcoin: A Peer-to-Peer Electronic Cash System. https://bitcoin.org/bitcoin.pdf
• Stuart Haber and W. Scott Stornetta. 1991. How to Time-Stamp a Digital Document. Journal of Cryptology 3, 2 (1991), 99–111. http://dx.doi.org/10.1007/bf00196791

-- Shawn M. Jones

Fig. 1: A screenshot of the landing page of an artifact on Figshare. Green boxes outline links to URIs that belong to this artifact. Red boxes outline links to that do not belong to this artifact.

Interestingly, this artifact links to another artifact, a master's thesis, that does not link back to this artifact, complicating discovery of the dataset associated with the research paper. Both artifacts are fully archived in the Internet Archive. In contrast, Fig. 2 below shows a different incomplete Figshare artifact -- as of April 12, 2017 -- at the Internet Archive. Through incidental crawling, the Internet Archive discovered the landing page for this artifact, but has not acquired the actual dataset or the bibliographic metadata. Such cases show that incidental crawling is insufficient for archiving scholarly artifacts.

Fig 2: A screenshot of the web pages from an incompletely archived artifact. The Internet Archive has archived the landing page of this Figshare artifact, but did not acquire the dataset or the bibliographic metadata about the artifact.

Fig. 3: A screenshot showing search engine results in Figshare that lead to entry pages for artifacts.

Fig. 4: This diagram demonstrates an artifact and its boundary. Artifacts often have links to content elsewhere in the scholarly portal,but only some of these links are to items that belong to the artifact.

Artifact Classes and Crawling Heuristics

Single Artifact URI and Single Hop

Fig 5a: A diagram of the Single Artifact URI artifact class.

Fig 5b: A diagram showing an example Single Hop artifact class.

Fig 5c: A diagram showing an example of a Multi-Hop artifact class.

1. Construct an invalid URI (i.e., one that produces a 404 HTTP status) for the portal.
2. Capture the content at that URI and place all links from that content into the ignored URIs list.
3. Capture the content from the homepage of the portal and place all links from that content into the ignored URIs list.
4. Remove the entry page URI from the ignored URIs list, if present.

Multi-Hop Artifacts

Fig. 6: Entry page belonging to the artifact representing the Memento Damage software project.

• https://github.com/erikaris/web-memento-damage/issues
• https://github.com/erikaris/web-memento-damage/graphs/contributors
• https://github.com/erikaris/web-memento-damage/blob/master/memento_damage/phantomjs/text-coverage.js
• https://github.com/erikaris/web-memento-damage/commit/afcdf74cc31178166f917e79bbad8f0285ae7831

Fig. 7: A diagram of a Dryad artifact consisting of a single dataset, but multiple metadata pages. Red boxes outline the significant string, Data from: Cytokine responses in birds challenged with the human food-borne pathogen Campylobacter jejuni implies a Th17 response., which is found in the title of the entry page and is present in almost all objects within the artifact. Only the dataset does not contain this significant string, hence a crawler must crawl URIs one hop out from those matching the Significant String from Title heuristic and also ignore menu items and other common links, hence the Significant String from Title+ with Ignored URIs is the prescribed heuristic in this case.

Fig. 8: A screenshot of an OSF artifact entry page showing the source in the lower pane. The title element of the page contains the string "OSF | Role of UvrD/Mfd in TCR Supplementary Material". The string "Role of UvrD/Mfd in TCR Supplementary Material" is present in all objects related to this artifact. To use this significant string, the substring  "OSF | " must be removed.

1. Capture the content of the entry page.
2. Save the text from the <title> tag of the entry page.
3. Capture the content of the portal homepage.
4. Save the text from the <title> tag of the homepage.
5. Starting from the leftmost character of each string, compare the characters of the entry page title text with the homepage title text.
1. If the characters match, remove the character in the same position from the entry page title.
2. Stop comparing when characters no longer match.

• http://www.gbif.org/occurrence/search?datasetKey=98333cb6-6c15-4add-aa0e-b322bf1500ba
• http://api.gbif.org/v1/dataset/98333cb6-6c15-4add-aa0e-b322bf1500ba/document

Comparison to Archive-It Settings

Fig. 9: This is a screenshot of part of the Archive-It configuration allowing a user to control the crawling strategy for each seed.

Fig. 10: This is a screenshot of the Archive-It configuration allowing the user to expand the crawl scope to include URIs that contain a given string.

Fig. 11: This screenshot displays the portion of the Archive-It configuration allowing the user to block URIs that contain a given string.

Heuristics Used In Review of Artifacts on Scholarly Portals

Further Thoughts

Conclusion

Building a global scholar profile

Scopus has multiple entries for "Herbert Van de Sompel".

Discovering Where Scholars Post Their Work

Homepages

Querying Portals Directly

1. Social Networking applies to portals that allow users to create connections to other users, usually via a "friend" network or by "following". Examples: MethodSpace, Twitter
2. Blogs encompasses everything from blog posts to magazine articles to forums. Examples: HASTAC, PubPeer, The Conversation
3. Content Communities involves portals where users share media, such as datasets, videos, and documents, including preprints. Examples: Figshare, BioRxiv, Slideshare, JoVe
4. Collaborative Works is reserved for portals where users collaboratively change a single produce, like a wiki page. Examples: Wikipedia

Local Portal Search Engines

This screenshot shows a common case of a search engine that provides profiles in its search results. I have outlined one of the links to a profile in a red box and shown a separate screenshot of the linked profile.

This screenshot shows an example of a search engine that does not provide profiles in its search results, even though the portal has profiles. The screenshots are of the search results, following the link to the document, and then following the link from the document to the profile page. Each followed link is outlined in a red box.

This screenshot shows an example of a site that does not provide user profiles at all, but does provide search results if a scholar's name shows up in a document.

Portal Web APIs

1. Look for the terms "developers", "API", "FAQ" on the main page of each portal. If present, follow those links to determine if the resulting resource contained further information on an API.
2. Use the local search engine to search for these terms
3. Use Google search with the following queries
1. site:<hostname> rest api
2. site:<hostname> soap api
3. site:<hostname> api
4. site:<hostname> developer
5. <hostname> api
6. <hostname> developer

Web Search Engines

Crawling the Portal and Building Our Own Search Engine