Why visualize individual web pages? A variety of visualizations of individual web pages exist, but why do we need them when we can just choose a URI from a list and put it in our web browser? URIs are intended to be opaque: text from the underlying web resource does not need to exist in the URI.

Consider http://dx.doi.org/10.1007/s00799-016-0200-8. Where does it go? Should we click on it? What content exists under the veil of the URI? Will it meet our needs?

Now consider this web page surrogate produced by embed.ly for the same URI:

Avoiding spoilers: wiki time travel with Sheldon Cooper

A variety of fan-based wikis about episodic fiction (e.g., television shows, novels, movies) exist on the World Wide Web. These wikis provide a wealth of information about complex stories, but if...

If we were looking for research papers about avoiding spoilers for TV shows, then we know that clicking on this surrogate will take us to something that meets our information needs. If we were searching for marine mammals, then this surrogate shows us that the underlying page will not be very satisfying. In this case, the surrogate is intended to give the user enough information to answer the question: should I click on this?

Last year, when I reviewed a number of live web curation and social media tools, I was primarily focused on tools that produce social cards like the one above. This was because social cards appeared to be the lingua franca of web page surrogates. Social cards are not the only surrogate in use today and definitely not the only surrogate evaluated in literature. In this post, I cover several surrogates that have been evaluated and then talk about the studies in which they played a part. I was curious as to which surrogate might be best for collections of mementos.

Different Web Page Surrogates

Text Snippet

Text snippets are one of the earliest surrogates. They only require fetching a given web page before selecting the text to be used in the snippet. The text selection can be done via many different methods like El-Beltagy's "KP-Miner: A Keyphrase Extraction System for English and Arabic Documents" and Chen's "A Practical System of Keyphrase Extraction for Web Pages". Text snippets are typically used by search engines for displaying results.

The Google search result text snippet for Michele Weigle's ODU CS page.

The Bing search result text snippet for Michele Weigle's ODU CS homepage. Note that Bing did not capture the last modified date, but does list a series of links on the bottom of the snippet, drawn from the menu of homepage

The DuckDuckGo search result for Michele Weigle's ODU CS homepage. Note that DuckDuckGo displays the favicon and generates a different text snippet from Google and Bing.

In the above search results for Michele Weigle's ODU CS homepage, the text snippets are slightly different depending on the search engine. Because there is a lot of variation in web pages, there are a lot of possibilities when building text snippets.

Text snippets still receive a bit of research, with Maxwell evaluating the effectiveness of snippet length in 2017 as part of "A Study of Snippet Length and Informativeness" (university repository copy).

As a group, text snippets listed one per row on a web page. This is optimal for search results, as the position of the result conveys its relevancy. This format affects how many surrogates can be viewed at once. Where text snippets are viewed one per row, more thumbnails can fit into the same amount of space.

Thumbnail

A thumbnail is produced by loading the given page in a browser and taking a screenshot of the contents of the browser window. They have been used in many forms. The Safari web browser uses them to display the content of tabs.

The Safari web browser uses thumbnails to show surrogates for web pages  that are currently loaded in its tabs.

In "Visual preview for link traversal on the World Wide Web", Kopetzky demonstrated that thumbnails could be used to provide a preview of a linked page via a mouseover effect so that users could decide if a link was worth clicking. In "Data Mountain: Using Spatial Memory for Document Management" (Microsoft Research copy), Robertson proposed using a 3D virtual environment for organizing a corpus of web pages where each page is visualized as a thumbnail. Outside of the web, file management tools, such as macOS's Finder, use thumbnails to provide visual previews of documents.

An example of the interface for Data Mountain, a 3D environment for browsing web pages via thumbnails.

macOS Finder displaying thumbnails of file contents.

In the web archiving world, the UK Web Archive uses thumbnails to show a series of mementos so one can compare the content of each memento, effectively viewing the content drift over time. Thumbnails are also used in our own What Did It Look Like?, a platform that animates thumbnails so one can watch the changes to a web page over the years. Our group is also investigating the use of thumbnails for summarizing how a single webpage has changed over time, using three different visualizations: an animation, a grid view, and an interactive timeline view.

The UK Web Archive uses thumbnails to show different mementos for the same resource, allowing the user to view web page changes over time.

What Did It Look Like? allows the user to watch a web page change over time by animating the thumbnails of the mementos of a resource.

The size of thumbnails has a serious effect on their utility. If the thumbnail is too large, it does not provide room for comparison of surrogates. If the thumbnail is too small, users cannot see what is in the image. Thumbnails are also difficult for users to understand if a page consists mostly of text or has no unique features. In "How People Recognize Previously Seen Web Pages from Titles, URLs and Thumbnails", Kaasten established that the optimal thumbnail size is 208x208 pixels.

The viewport of a thumbnail is also an important part of its construction. Depending on what we want to emphasize on a web page, we may need to generate a thumbnail from content "below the fold". Aula evaluated the use of thumbnails that were the same size, but had magnified a portion of a web page at 20% versus 38%. She found that users performed better with thumbnails at a magnification of 20%.

Enhanced Thumbnail

In 2001, Woodruff introduced the enhanced thumbnail in "Using Thumbnails to Search the Web" (author copy). Prior to taking the screenshot of the browser as with a normal thumbnail, the HTML of the page is modified to make certain terms stand out. In the example below, changes in font size and background color emphasize certain terms of a page. The goal is to draw attention to these terms in hopes that search engine users could find relevant pages faster.

Examples of Thumbnails and Enhanced Thumbnails: (a) Plain thumbnail (b) Enhanced Thumbnail using HTML modification to emphasize the words "Recipe" and "Pound Cake" (c) Enhanced Thumbnail using HTML and image modification to make "Recipe" and "Pound Cake" stand out more (d) Emphasis on "MiniDisc Player" (e) Emphasis on "hybrid", "car", and "mileage" (f) Emphasis on "Hellerstein" (g) Plain thumbnail of a page only consisting of text (h) Enhanced thumbnail emphasizing specific terms in the text page

Even though enhanced thumbnails have performed well, they are computationally expensive to create. This likely explains why they have not been seen in use outside of laboratory studies.

In "Evaluating the Effectiveness of Visual Summaries for Web Search", Al Maqbali developed something similar by adding a tag cloud to each thumbnail and named the concept a "visual tag".

Internal Image

An internal image is an image embedded within the web page. For some web pages, like news stories and product pages, these internal images can be good surrogates because of their uniqueness. Pinterest uses internal images as surrogates.

Pinterest uses internal images as surrogates for web pages.

The key is identifying which embedded image is best for representing the page. Hu identified the issues with solving this problem as part of "Categorizing Images in Web Documents", identifying a number of features such as using the text surrounding an image and evaluating the number of colors in the image. Maekawa worked on classifying images and achieved an 83.1% accuracy in "Image Classification for Mobile Web Browsing" (conference copy). While these studies provided solutions for classifying images, we really need to know which images are unique and relevant to the web page. Research does exist to address this issue, such as the work described in Li's "Improving relevance judgment of web search results with image excerpts" (conference copy). These solutions are imperfect, which may be why Pinterest and other sites ask the user to choose an image from those embedded in the page.

Visual Snippet

In 2009, Teevan introduced visual snippets as part of "Visual snippets: summarizing web pages for search and revisitation" (Microsoft Research copy, conference slides). Teevan gave 20 web pages to a graphic designer and asked him to generate a small 120x120 image representing each page. She observed a pattern in the resulting images and derived a template to use as a surrogate. These surrogates combine the internal image, placed within the background of the surrogate, with a title running across the top of the page, and a page logo.

Examples of thumbnails on the bottom and their corresponding visual snippets on top.

She used machine learning to choose a good internal image and logo. This is more complex than merely selecting a salient internal image as noted in the previous section. Not only does the visual snippet require two images, but two different types of images.

External Image

In 2010, Jiao put forth the idea of using external images in "Visual Summarization of Web Pages". Jiao notes that detecting the internal image may be difficult if not impossible for some pages. Instead, he suggests using image search engines to find a representative image to use as a surrogate.

A simplified version of his algorithm is:

1. Extract key phrases from the target web page using Chen's KEX algorithm
2. Use these phrases as queries for an image search engine
3. Rerank the search engine results based on textual similarity to the target web page
4. Choose the top ranked image

Though this would likely work well for live web pages about products, it may be a poor fit for mementos due to the temporal nature of words. Consider a memento from the late 1990s where one of the key phrases extracted contains the word Clinton. In the 1990s, the document was likely referring to US President Bill Clinton. If we use a search engine in 2018, it may return an image of 2016 presidential candidate Hillary Clinton. Some of these temporal issues have been detailed as part of the Longitudinal Analytics on Web Archive Data (LAWA) project.

Text + Thumbnail

In "A Comparison of Visual and Textual Page Previews in Judging the Helpfulness of Web Pages" (google research copy) by Aula and "Do Thumbnail Previews Help Users Make Better Relevance Decisions about Web Search Results?" by Dziadosz, the authors consider the combination of text with a thumbnail as a surrogate.

The Internet Archive uses text and thumbnails for its search results, seen in the screenshot below.

The Internet Archive uses thumbnails and text together as part of its search results.

Al Maqbali further extended this concept with a text + visual tags.

Social Card

<meta name="twitter:card" content="summary"><meta name="twitter:site" content="@shawnmjones"><meta name="twitter:image:src" content="http://www.shawnmjones.org/images/20_me.jpg"><meta name="twitter:card" content="summary"><meta name="twitter:creator" content="@shawnmjones"><meta name="twitter:title" content="Shawn M. Jones"><meta name="twitter:description" content="Everything you need to know about Shawn M. Jones">

Social card for https://www.shawnmjones.org as seen on Twitter.

For Facebook, I used the following tags to produce the card below: <meta property="og:type" content="website"><meta property="og:title" content="Shawn M. Jones"><meta property="og:image" content="http://www.shawnmjones.org/images/20_me.jpg"><meta property="og:url" content="http://www.shawnmjones.org/"><meta property="og:site_name" content="Shawn M. Jones"><meta property="og:locale" content="en_US"><meta property="og:description" content="Description of the professional life of Shawn M. Jones">

Social card for https://www.shawnmjones.org as seen on Facebook.

Note how the HTML tags are different for each service. Facebook supports the Open Graph Protocol, developed around 2009 (according to the CarbonDate service) whereas Twitter's features were developed around 2010 (according to CarbonDate). There are pages that lack this kind of assistive markup. To produce those cards, social media platforms will often use other methods, like those mentioned above, to extract a text snippet and an internal image. Any mementos captured prior to 2009 will not have the benefit of this assistive markup.

Though most platforms generate social cards come in landscape form, some do generate a portrait form as well. The intended use of the social cards on the platform and the nature of other visual cues on the platform often drive the decision as to which form the social card should take. All of the studies in this blog post evaluated social cards in their landscape form.

A landscape social card from Facebook.

A portrait social card from Google+.

Social cards are not just used by social media. Wikipedia uses social cards to provide a preview of links if the user hovers over the link, like what Kopetzky had envisioned with thumbnails. Google News often uses social cards for individual stories. Social cards sometimes include additional information beyond text snippet and image. In "What's Happening and What Happened: Searching the Social Web" Omar Alonso detailed the use of social cards in a prototype for Bing search results. Those cards also incorporated lists of users who shared the target web page as well as associated hashtags.

When a user hovers over an internal link, Wikipedia uses social cards  to display a preview of the linked web page.

Google News often uses social cards to list individual news articles.

There are similar concepts that are not instances of the social card. Some of the cards used by Google News are not social cards because each is a surrogate for a news story spanning multiple resources, rather than a single resource. Likewise, search engines use entity cards to display information about a specific entity drawn from multiple sources. Entity cards have been found to be useful by Bota's 2016 study "Playing Your Cards Right: The Effect of Entity Cards on Search Behaviour and Workload". I do not consider entity cards to be social cards because each social card is a surrogate for a single web resource, whereas an entity card is a surrogate for a conceptual entity and is drawn from multiple sources.

This card used by Google News is not a surrogate for a single web resource, and hence I do not consider it a social card.

This card format, used by Google is also not a surrogate for a single web resource. This is an entity card, drawing from multiple web resources.

The creation of social cards can also be a lucrative market, with Embed.ly offering plans for web platforms ranging from $9 to $99 per month. They provide embedding services for the long form blogging service Medium, supporting a limited number of source websites. Individual cards can be made on their code generation page.

Evaluations of these Surrogates

Web page surrogates have been of great interest to those studying search engine result pages. I have review eight studies on web surrogates, most mentioned above. I focused on how these studies compared surrogates with each other.

Author & Year	Text Snippet	Internal/ External Image	Visual Snippet	Thumbnail	Enhanced Thumbnail/ Visual Tags	Text + Thumbnail	Social Card
Woodruff 2001	X			X	X
Dziadosz 2002	X			X		X
Li 2008	X						X
Teevan 2009	X		X	X
Jiao 2010		X	X	X
Aula 2010	X			X		X
Al Maqbali 2010	X		X		X	X	X
Loumakis 2011	X	X					X
Capra 2013	X	X					X

As noted above Woodruff introduced the concept of enhanced thumbnails in "Using Thumbnails to Search the Web". To evaluate their effectiveness, she generated questions based on tasks users commonly perform on the web. The questions were divided into 4 categories and 3 questions per category were each given to 18 participants. The participants were presented with search engine result pages consisting of 100 text snippets, thumbnails, or enhanced thumbnails. In their attempt to find web resources that would address their assigned questions, participants were evaluated based on their response times. The results indicated that enhanced thumbnails provided the fastest response times overall, but the results varied depending on the type of task. For locating an entity's homepage, text snippets and enhanced thumbnails performed roughly the same. For finding the picture of an entity, thumbnails and enhanced thumbnails performed roughly the same. All three surrogate types performed just as well for e-commerce or medical side-effect questions.

Dziadosz tested the concept of text snippets combined with thumbnails in "Do Thumbnail Previews Help Users Make Better Relevance Decisions about Web Search Results?" In this study of 35 participants, each was given 2 queries each and 2 tasks. Each participant was given a different surrogate type. Their first task was to identify all search engine results on the page that they assumed to be relevant to their query. Their second task was to visit the pages being the surrogates and identify which were actually relevant. The number of correct decisions for text snippets combined with thumbnails was higher than just for text or just for thumbnails. Aula, in "A Comparison of Visual and Textual Page Previews in Judging the Helpfulness of Web Pages" also evaluated text snippets, thumbnails, and a combination. She discovered that both were effective in making relevance judgements.

Teevan evaluated the effectiveness of visual snippets in "Visual snippets: summarizing web pages for search and revisitation". Her study consisted on 276 participants who were each given 12 search tasks and a set of 20 search results, with 4 of the 12 tasks completed with different surrogates. She discovered that text snippets required the fewest clicks compared to thumbnails, which required the most. This indicates a lot of false positive matches for participants when using thumbnails. Participants preferred visual snippets or text snippets equally over thumbnails and preferred visual snippets for shopping tasks. Most participants found thumbnails to be too small to be useful.

Jiao introduced the concept of using external images as a surrogate in "Visual Summarization of Web Pages". He compared the use of internal images, external images, thumbnails, and visual snippets. Like Dziadosz's study, participants were asked to guess the relevance of the web page behind the surrogate and then later evaluate if their earlier guess was correct. To generate search results, they randomly sampled 100 queries from the KDD CUP '05 dataset and submitted them to Bing. His results show that none of the surrogates works for all types of pages. Overall internal images were best for pages that contained a dominant image whereas thumbnails or external images were best for understanding pages that did not contain a dominant image.

In "Improving relevance judgment of web search results with image excerpts", Li was interested in identifying dominant images in web pages. I focus here on the second study in his work which compares text snippets and social cards. They randomly sampled 100 queries from the KDD CUP '05 dataset and submitted them to Google. The search engine results were then evaluated and reformatted into either text snippets or social cards. Two groups of 12 students each were given the queries either classified by their functionalities or semantic categories. The participants were evaluated based on the number of clicks of relevant results and also on the amount of time they took with each search. Social cards were the clear winner over text snippets in terms of time and clicks.

Loumakis, in "This Image Smells Good: Effects of Image Information Scent in Search Engine Results Pages" (university copy) attempted to compare the performance of images, text snippets, and social cards. Using preselected queries and 81 participants, Loumakis also reformatted Google search results. He did not get the same level of performance in his study, noting that "Adding an image to a SERP result will not significantly help users in identifying correct results, but neither will it significantly hinder them if an image is placed with text cues where the scents may conflict."

In "Evaluating the Effectiveness of Visual Summaries for Web Search", Al Maqbali explored the use of different image augmentations for visual snippets, text + thumbnail, social card, text + visual snippet, and a text + tag cloud/thumbnail combination. Al Maqbali had 65 participants evaluate the relevance of search engine result pages as in the prior studies. This study reached the same conclusion as Loumakis: adding images to text snippets does not appear to make a difference to the performance of search engine users.

To further understand the disagreement between the results of Loumakis, Al Maqbali, and Li, in "Augmenting web search surrogates with images", Capra explored the effectiveness of text snippets and social cards. He wanted determine if the quality or relevance of the image used in the social card had any effect on performance. Prior to any relevance study, he had one set of participants rate individual internal images for a social card as good, bad, and mixed. For individual surrogates, Capra discovered that text snippets with good images have a slightly higher statistically significant accuracy score than just text snippets alone, at the cost of judgement duration for each surrogate. The accuracy for text snippets was 0.864, the accuracy for social cards with bad images was also 0.864, and the accuracy for social cards with good images was 0.884. If the search engine result pages were evaluated overall, then there was evidence that good images showed improvement in accuracy with ambiguous queries (e.g., jaguar the car or the cat?), but in this case the improvements were not statistically significant.

Deciding on the best surrogate for use with web pages depends on a number of factors, and the studies comparing these surrogates have some disagreement. Text snippets continue to endure for search results likely due to Capra's, Al Maqbali's, and Loumakis' results. Social cards are preferred by users, but the minor improvement in search time and relevance accuracy does not warrant the effort necessary to select a good internal image for the card. This means that social cards are effectively relegated for use in social media where each can be generated individually rather than with hundreds of search results. This also means that thumbnails are relegated to other tasks, such as a surrogate for a file on a filesystem or within a browser's interface. As most of these studies focused primarily on search engine results, it is likely that many of these surrogates work better with other use cases.

Surrogates for Mementos

There are more uses for surrogates than search engine results. When grouped together, some surrogates provide more information than the answer to the question should I click on this?.

Enhanced thumbnails often reflect the search terms of the query provided by the user. Most memento applications do not have a query, and hence there are not words or phrases to enhance within the thumbnail. Mabali's tag cloud concept may be of interest here. I am examining other ways to expose words and phrases of interest from archived collections, so this surrogate type may find new life in mementos.

Internal images are often used as part of social cards. If one could expose the images that tie to a particular theme in a web archive collection, then it is possible that we could select images for use as memento surrogates within the theme of the collection. This would likely require some form of machine learning to be viable. This same process goes for visual snippets.

As noted above, external images are problematic surrogates for mementos due to the temporal nature of words. If we could divide a web archive into specific time periods, then external images could be extracted from pages around the same time, limiting the amount of temporal shift.

Thumbnails are often useful in groups to demonstrate the content drift of a single web resource. For this surrogate group to be useful, the consumer of such a thumbnail gallery needs to understand the direction that time flows in the visualization. Thumbnails are not limited to the "one-per-row" paradigm of landscape social cards or text snippets, and hence thumbnails can be presented in a grid formation. This can be confusing to the user trying to compare the content drift of a resource, but textual cues, such as the memento-datetime, placed above or below the thumbnail can clear up this confusion.

Storytelling often uses surrogates in the form of social cards to tell a story. In this case, the surrogates are visualizations of the underlying web pages. When provided as a series of social cards, one per row, in order to publication date or memento-datetime, collections of these surrogates can convey information about an unfolding news story, such as in AlNoamany's collection summarization work (preprint version, dissertation). Many mementos do not have the metadata that might assist in finding a good internal image. This means that any service providing social cards to mementos must instead rely upon a number of image selection algorithms with differing levels of success. Because text snippets are essentially social cards lacking an image, is it possible that they, too, would be suitable in this context?

Conclusion

I started on this journey looking for the best surrogate for use with mementos. I discovered many different surrogates for web resources. The studies evaluating these different surrogates focused on the success of users finding relevant information in search engine results. It appears that the search engine industry has largely focused on text snippets as they are the least expensive surrogate to produce and studies indicate that the addition of images has minimal impact on their effectiveness. Mementos have many different uses and it is possible that one or more of these surrogates may be better fit for their temporal nature. Now that I am developing a vocabulary for these surrogates, I can start to explore how they might best be used with mementos, bringing other useful visualizations to web archive collections.

-- Shawn M. Jones

This past December, Reid’s lawyers contacted us, asking to have archives of the blog (blog.reidreport.com) taken down, stating that “fraudulent” posts were “inserted into legitimate content” in our archives of the blog. Her attorneys stated that they didn’t know if the alleged insertion happened on the original site or with our archives (Reid’s claim regarding the point of manipulation is still unclear to us).

...

At some point after our correspondence, a robots.txt exclusion request specific to the Wayback Machine was placed on the live blog. That request was automatically recognized and processed by the Wayback Machine and the blog archives were excluded, unbeknownst to us (the process is fully automated). The robots.txt exclusion from the web archive remains automatically in effect due to the presence of the request on the live blog.   

2006-01-20

30/x Joy seemed very interested in Brokeback Mountain, but wouldn't watch it bc it featured two men hooking up.
 
She can't understand who is going to see it since she imagines everyone would be turned off by it. pic.twitter.com/ogBnPyDSUF

— Not a bot (@Jamie_Maz) April 18, 2018

2006-06-05

2006-06-13

2006-10-03

2008-10-16

It is hard to believe that the time has come for me to write a wrap up blog about the adventure that was my Masters Degree and the thesis that got me to this point. If you follow this blog with any regularity you may remember two posts, written by myself, that were the genesis of my thesis topic:

• 2017-01-20: CNN.com has been unarchivable since November 1st, 2016
• 2017-03-09: A State Of Replay or Location, Location, Location

Bonus points if you can guess the general topic of the thesis from the titles of those two blog posts. However, it is ok if you can not as I will give an oh so brief TL;DR;. The replay problems with cnn.com were, sadly, your typical here today gone tomorrow replay issues involving this little thing, that I have come to , known as JavaScript. What we also found out, when replaying mementos of cnn.com from the major web archives, was each web archive has their own unique and subtle variation of this thing called "replay". The next post about the curious case of mendely.com user pages (A State Of Replay) further confirmed that to us.

We found that not only does there exist variations in how web archives perform URL rewriting (URI-Rs URI-Ms) but also that, depending on the replay scheme employed, web archives are also modifying the JavaScript execution environment of the browser and the archived JavaScript code itself beyond URL rewriting! As you can imagine this left us asking a number of questions that lead to the realization that the web archiving lacks the terminology required to effectively describe the existing styles of replay and the modifications made to an archived web page and its embedded resources in order to facilitate replay.

Thus my thesis was born and is titled "To Relive The Web: A Framework For The Transformation And Archival Replay Of Web Pages".

• Defense slides ( with animations)
• Defense recording

Since I am known around the ws-dl headquarters for my love of deep diving into the secrets of (securely) replaying JavaScript, I will keep the length of this blog post to a minimum. The thesis can be broken down into three parts, namely Styles Of Replay, Memento Modifications, and Auto-Generating Client-Side Rewriters. For more detail information about my thesis, I have embedded my defense's slides below and the full text of the thesis has been made available.

Styles Of Replay

The existing styles of replaying mementos from web archives is broken down into two distinct models, namely "Wayback" and "Non-Wayback", and each has its own distinct styles. For the sake of simplicity and length of this blog post I will only (briefly) cover the replay styles of the "Wayback" model.

Non-Sandboxing Replay

Non-sandboxing replay is the style of replay that does not separate the replayed memento from the archive-controlled portion of replay, namely the banner. This style of replay is considered the OG (original gangster) way for replaying mementos simply because it was, at the time, the only way to replay mementos and was introduced by the Internet Archive's Wayback Machine. To both clarify and illustrate what we mean by "does not separate the replayed memento from archive-controlled portion of replay", consider the image below displaying the HTML and frame tree for a http://2016.makemepulse.com memento replayed from the Internet Archive on October 22, 2017.

As you can see from the image above, the archive's banner and the memento exist together on the same domain (web.archive.org). Implying that the replayed memento(s) can tamper with the banner (displayed during replay) and or interfere with archive control over replay. Non-malicious examples of mementos containing HTML tags that can both tamper with the banner and interfere with archive control over replay skip to the Replay Preserving modifications section of post. Now to address the recent claim that "memento(s) were hacked in the archive" and its correlation to non-sanboxing replay. Additional discussion on this topic can be found in Dr. Michael Nelson's blog post covering the case of blog.reidreport.com and in his presentation for the National Forum on Ethics and Archiving the Web (slides, trip report).

For a memento to be considered (actually) hacked, the web archive the memento is replayed (retrieved) from must be have been compromised in a manner that requires the hack to be made within the data-stores of the archive and does not involve user initiated preservation. However, user initiated preservation can only tamper with a non-hacked memento when replayed from an archive. The tampering occurs when an embedded resource, previously un-archived at the memento-datetime of the "hacked" memento, is archived from the future (present datetime relative to memento-datetime) and typically involves the usage of JavaScript. Unlike non-sandboxing replay, the next style of Wayback replay, Sandboxed Replay, directly addresses this issue and the issues of how to securely replay archived JavaScript. PS. No signs of tampering, JavaScript based or otherwise, were present in the blog.reidreport.com mementos from the Library of Congress. How do I know??? Read my thesis and or look over my thesis defense slides, I cover in detail what is involved in the mitigation of JavaScript based memento tampering and know what that actually looks like .

Sandboxed Replay

Sandboxed replay is the style of replay that separates the replayed memento from the archive-controlled portion of the page through replay isolation. Replay isolation is the usage of an iframe to the sandbox the replayed memento, replayed from a different domain, from the archive controlled portion of replay. Because replay is split into two different domains (illustrated in the image seen below), one for the replay of the memento and one for the archived controlled portion of replay (banner), the memento cannot tamper with the archives control over replay or the banner. Due to security restrictions placed on web pages from different origins by the browser called the Same Origin Policy. Web archives employing sandboxed replay typically also perform the memento modification style known as Temporal Jailing. This style of replay is currently employed by Webrecorder and all web archives using Pywb (open source, python implementation of the Wayback Machine). For more information on the security issues involved in high-fidelity web archiving see the talk entitled Thinking like a hacker: Security Considerations for High-Fidelity Web Archives given by Ilya Kreymer and Jack Cushman at WAC2017 (trip report), as well as, Dr. David Rosenthal's commentary on the talk.

Memento Modifications

The modification made by web archives to mementos in order to facilitate there replay can be broken down into three categories, the first of which is Archival Linkage.

Archival Linkage Modifications

Archival linkage modifications are made by the archive to a memento and its embedded resources in order to serve (replay) them from the archive. The archival linkage category of modifications are the most fundamental and necessary modifications made to mementos by web archives simply because they prevent the Zombie Apocalypse. You are probably already familiar with this category of memento modifications as it is more commonly referred to as URL rewriting
(URI-R URI-M).

<!-- pre rewritten -->
<link        class="token attr-name">rel=        class="token punctuation">"stylesheet"        class="token attr-name">href=        class="token punctuation">"/foreverTime.css"        class="token punctuation">>
<!-- post rewritten -->
<link        class="token attr-name">rel=        class="token punctuation">"stylesheet"        class="token attr-name">href=        class="token punctuation">"/20171007035807cs_/foreverTime.css        class="token punctuation">">

URL rewriting (archival linkage modifications) ensures that you can relive (replay) mementos, not from the live web, but from the archive. Hence the necessity and requirement for this kind of memento modifications. However, it is becoming necessary to seemingly damage mementos in order to simply replay them.

Replay Preserving Modifications

Replay Preserving Modifications are modifications made by web archives to specific HTML element and attribute pairs in order to negate their intended semantics. To illustrate this, let us consider two examples, the first of which was introduced by our fearless leader Dr. Michael Nelson and is known as the zombie introducing meta refresh tag shown below.

        class="token punctuation"><metahttp-equiv        class="token attr-value">="refresh        class="token punctuation">"content        class="token attr-value">="35;url=?zombie=666        class="token punctuation">"/>

As you are familiar, the meta refresh tag will, after 35 seconds, refresh the page with the "?zombie=666" appended to original URL. When a page containing this dastardly tag is archived and replayed, the results of the refresh plus appending "?zombie=666" to the URI-M causes the browser to navigate to a new URI-M that was never archived. To overcome this archives must arm themselves with the attribute prefixing shotgun in order to negate the tag and attribute's effects. A successful defense against the zombie invasion when using the attribute prefixing shotgun is shown below.

            class="token punctuation"><meta_http-equiv            class="token attr-value">="refresh            class="token punctuation">"_content            class="token attr-value">="35;url=?zombie=666            class="token punctuation">"/>

Now let me introduce to you a new more insidious tag that does not introduce a zombie into replay but rather a demon known as the meta csp tag, shown below.

        class="token punctuation"><metahttp-equiv        class="token attr-value">="Content-Security-Policy        class="token punctuation">"
content            class="token punctuation">="default-src http://notarchive.com; img-src ....            class="token punctuation">"/>

Naturally, web archives do not want web pages to be delivering their own Content-Security-Policies via meta tag because the results are devastating, as shown by the YouTube video below.

Readers have no fear, this issue is fixed!!!! I fixed the meta csp issue for Pywb and Webrecorder in pull request #274 submitted to Pywb. I also reported this to the Internet Archive and they promptly got around to fixing it.

Temporal Jailing

The final category of modifications, known as temporal Jailing, is the emulation of the JavaScript environment as it existed at the original memento-datetime through client-side rewriting. Temporal jailing ensures both the secure replay of JavaScript and that JavaScript can not tamper with time (introduce zombies) by applying overrides to the JavaScript APIs provided by the browser in order to intercept un-rewriten urls. Yes there is more to it, a whole lot more, but because it involves replaying JavaScript and I am attempting to keep this blog post reasonably short(ish), I must force you to consult my thesis or thesis defense slides for more specific details. However, for more information about the impact of JavaScript on archivability, and measuring the impact of missing resources see Dr. Justin Brunelle'sPh.D. wrap up blog post. The technique for the secure replay of JavaScript known as temporal jailing is currently used by Webrecorder and Pywb.

Auto-Generating Client-Side Rewriters

Have I mention yet just how much I JavaScript?? If not, lemme give you a brief overview of how I am auto-generating client-side rewriting libraries, created a new way to replay JavaScript (currently used in production by Webrecorder and Pywb) and increased the replay fidelity of the Internet Archive's Wayback Machine.

First up let me introduce to you Emu: Easily Maintained Client-Side URL Rewriter (GitHub). Emu allows for any web archive to generate their own generic client-side rewriting library, that conforms to the de facto standard implementation Pywb's wombat.js, by supplying it the Web IDL definitions for the JavaScript APIs of the browser. Web IDL was created by the W3C to describe interfaces intended to be implemented in web browser, allow the behavior of common script objects in the web platform to be specified more readily, and provide how interfaces described with Web IDL correspond to constructs within ECMAScript execution environments. You may be wondering how can I guarantee this tool will generate a client-side rewriter that provides complete coverage of the JavaScript APIs of the browser and that we can readily obtain these Web IDL definitions? My answer is simple and it is to confider the following excerpt from the HTML specification:

This specification uses the term document to refer to any use of HTML, ..., as well as to fully-fledged interactive applications. The term is used to refer both to Document objects and their descendant DOM trees, and to serialized byte streams using the HTML syntax or the XML syntax, depending on context ... User agents that support scripting must also be conforming implementations of the IDL fragments in this specification, as described in the Web IDL specification

Pretty cool right, what is even cooler is that a good number of your major browsers/browser engines (Chromium, FireFox, and Webkit) generate and make publicly available Web IDL definitions representing the browsers/engines conformity to the specification! Next up a new way to replay JavaScript.

Remember the curious case of mendely.com user pages (A State Of Replay) and how we found out that Archive-It, in addition to applying archival linkage modifications, was rewriting JavaScript code to substitute a new foreign, archive controlled, version of the JavaScript APIs it was targeting. This is shown in the image below.

Archive-It rewriting embedded JavaScript from the memento for the curious case mendely.com user pages

Hmmmm, looks like Archive-It is only rewriting only two out of four instances of the text string location in the example shown above. This JavaScript rewriting was targeting the Location interface which controls the location of the browser. Ok, so how well would Pywb/Webrecorder do in this situation?? From the image shown below, not as good and maybe a tad bit worse...

Pywb v0.33 replay of https://reacttraining.com/react-router/web/example/auth-workflow

That's right folks, JavaScript rewrites in HTML. Why??? See below.

Bundling HTML in JavaScript, https://reacttraining.com/react-router/15-5fae8d6cf7d50c1c6c7a.js

Because the documentation site for React Router was bundling HTML inside of JavaScript containing the text string "location" (shown above), the rewrites were exposed in the documentations HTML displayed to page viewers (second image above). In combination with how Archive-It is also rewriting archived JavaScript, in a similar manner, I was like this needs to be fix. And fix it I did. Let me introduce to you a brand new way of replaying archived JavaScript shown below.

// window proxy
newwindow.Proxy            class="token punctuation">({            class="token punctuation">},            class="token punctuation">{
get(target            class="token punctuation">, prop)            class="token punctuation">{            class="token comment">/*intercept attribute getter calls*/            class="token punctuation">},
set(target            class="token punctuation">, prop, value            class="token punctuation">){/*intercept attribute setter calls*/            class="token punctuation">},
has(target, prop            class="token punctuation">){/*intercept attribute lookup*/            class="token punctuation">},
ownKeys(target            class="token punctuation">){/*intercept own property lookup*/            class="token punctuation">},
getOwnPropertyDescriptor(target            class="token punctuation">, key)            class="token punctuation">{/*intercept descriptor lookup*/            class="token punctuation">},
getPrototypeOf(target            class="token punctuation">){/*intercept prototype retrieval*/            class="token punctuation">},
setPrototypeOf(target            class="token punctuation">, newProto)            class="token punctuation">{/*intercept prototype changes*/            class="token punctuation">},
isExtensible(target            class="token punctuation">){/*intercept is object extendable lookup*/            class="token punctuation">},
preventExtensions(target            class="token punctuation">){/*intercept prevent extension calls*/            class="token punctuation">},
deleteProperty(target            class="token punctuation">, prop)            class="token punctuation">{/*intercept is property deletion*/            class="token punctuation">},
defineProperty(target            class="token punctuation">, prop, desc            class="token punctuation">){/*intercept new property definition*/            class="token punctuation">},
})

// document proxy
newwindow.Proxy            class="token punctuation">(window.document            class="token punctuation">,{
get(target            class="token punctuation">, prop)            class="token punctuation">{            class="token comment">/*intercept attribute getter calls*/            class="token punctuation">},
set(target            class="token punctuation">, prop, value            class="token punctuation">){/*intercept attribute setter calls*/            class="token punctuation">}
})

The native JavaScript Proxy object allows an archive to perform runtime reflection on the proxied object. Simply put, it allows an archive to defined custom or restricted behavior for the proxied object. I have annotated the code snippet above with additional information about the particulars of how archives can use the Proxy object. Archives using the JavaScript Proxy object in combination with the setup shown below, web archives can guarantee the secure replay of archived JavaScript and do not have to perform the kind of rewriting shown above. Yay! Less archival modification of JavaScript!! This method of replaying archived JavaScript was merged into Pywb on August 4, 2017 (contributed by yours truly) and has been used in production by Webrecoder since August 21, 2017. Now to tell you about how I increased the replay fidelity of the Internet Archive and how you can too .

        class="token keyword">var        class="token function-variable function">__archive$assign$function__        class="token operator">=function        class="token punctuation">(name)        class="token punctuation">{/*return archive override*/        class="token punctuation">};
{
// archive overrides shadow these interfaces
let window =__archive$assign$function__            class="token punctuation">("window")            class="token punctuation">;
let self =__archive$assign$function__            class="token punctuation">("self"            class="token punctuation">);
let document =__archive$assign$function__            class="token punctuation">("document"            class="token punctuation">);
let location =__archive$assign$function__            class="token punctuation">("location"            class="token punctuation">);
let top =__archive$assign$function__            class="token punctuation">("top"            class="token punctuation">);
let parent =__archive$assign$function__            class="token punctuation">("parent")            class="token punctuation">;
let frames =__archive$assign$function__            class="token punctuation">("frames")            class="token punctuation">;
let opener =__archive$assign$function__            class="token punctuation">("opener")            class="token punctuation">;
/* archived JavaScript */
}

Ok so I generated a client-side rewriter for the Internet Archive's Wayback Machine using the code that is now Emu and crawled 577 Internet Archive mementos from the top 700 web pages found in the Alexa top 1 million web site list circa June 2017. The crawler I wrote for this can be found on GitHub . By using the generated client-side rewriter I was able to increase the cumulative number of requests made by the Internet Archive mementos by 32.8%, a 45,051 request increase (graph of this metric shown below). Remember that each additional request corresponds to a resource that previously was unable to be replayed from the Wayback Machine.

Hey look, I also decreased the number of requests blocked by the content-security policy of the Wayback Machine by 87.5%, a 5,972 request increase (graph of this metric shown below). Remember, that earch request un-blocked corresponds to a URI-R the Wayback Machine could not rewrite server-side and requires the usage of client-side rewriting (Pywb and Webrecorder are using this technique already).

Now you must be thinking this impressive to say the least, but how do I know these numbers are not faked / or doctored in some way in order to give a client-side rewriting the advantage??? Well you know what they say seeing is believing!!! The generated client-side rewriter used in the crawl that produced the numbers shown to you today is available as the Wayback++ Chrome and Firefox browser extension! Source code for it is on GitHub as well. And oh look, a video demonstrating the increase in replay fidelity gained if the Internet Archive were to use client-side. Oh, I almost forgot to mention that at the 1:47 mark in the video I make mementos of cnn.com replayable again from the Internet Archive. Winning!!

Pretty good for just a masters thesis wouldn't you agree. Now it's time for the obligatory list of all the things I have created in the process of this research and time as a masters student:

• Squidwarc: A high fidelity archival crawler that uses Chrome or Chrome Headless (blog post)
• MS Thesis Crawler: The Chrome or Chrome Headless based crawler written for my thesis's evaluation to specifically crawl the Internet Archive
• Wayback++: A Chrome and Firefox browser extension that brings client-side rewriting to the Internet Archive's Wayback Machine
• Emu: Easily Maintained Client-Side URL Rewriter. Generate a client-side rewriter from Web IDL
• Explore the evaluation dataset: a site built to allow access to the dataset of my thesis's evaluation
• node-cdxj: Read cdxj files produced by Pywb using node.js (npm)
• node-warc: Parse Web Archive (WARC) files or create WARC files using Electron or the chrome-remote-interface. (npm)
• Memgator Bulk TimeMap Downloader: Have you ever had a need to download 100 or 1 million TimeMaps using Memgator? With the caveat that it must be done in a timely manner? If so then you are in luck because this project has you covered.
• Grad School Python Utils: A python 3 utility belt containing a collection of reusable python code to aid grad students in getting through grad school
• Latex Toolbox: Librarification of the setup and macros used in the Latex src of my masters thesis
• WAIL Electron: An updated, Electron version of the Mat Kelly original production WAIL Python (Web Archiving Interface Layer)
  • Blog posts: 1, 2
  • JCDL2017: Paper, BibTex
• Auto Check Thy Margins: A 2018 re-implementation of the make file created by Dr. Chuck Cartledge (original blog post)
• User-Agent Lists: 12,296 User-Agent strings for research purposes only of course
• nukeBloggerClickTrap.js: A small, embeddable, and vanilla JavaScript code that will remove the annoying blogger click trap element when previewing your blog post
• Swimming In A Sea Of JavaScript, Or: How I Learned To Stop Worrying And Love High-Fidelity Replay, Accepted WADL 2018 submission
• To Relive The Web: A Framework For The Transformation And Archival Replay Of Web Pages,
  • Defense slides ( with animations)
  • Defense recording

What is next you may ask??? Well I am going to be taking a break before I start down the path known as a Ph.D. Why??????? To become the senior backend developer for Webrecorder of course! There is so so much to be learned from actually getting my hands dirty in facilitating high-fidelity web archiving such that when I return, I will have a much better idea of what my research's focus should be on.

If I have said this once, I have said this a million times. When you use a web browser in the preservation process, there is no such thing as an un-archivable web page! Long live high-fidelity web archiving!

Fig. 2: WSDL URL diversity matrix of examples across multiple policies (URL, hostname, and domain). For all policies, the schemes, URL parameters, and fragments are stripped before calculation. For hostname diversity calculation, only the host is considered, and for domain diversity calculation, only the domain is considered.

Fig. 3: The WSDL diversity index (Equation 1) and the explanation of variables. U represents the count of unique URLs (or species - R).  |C| represents the number of URLs (or individuals N).

The Archives Unleashed team (pictured below) hosted a two-day datathon, April 26-27, 2018, at the University of Toronto’s Robarts Library. This time around, Shawn Jones and I were selected to represent the Web Science and Digital Libraries (WSDL) research group from Old Dominion University. This event was the first in a series of four planned datathons to give researchers, archivists, computer scientists, and many others the opportunity to get hands-on experience with the Archives Unleashed Toolkit (AUT) and provide valuable feedback to the team. The AUT facilitates analysis and processing of web archives at scale and the datathons are designed to help participants find ways to incorporate these tools into their own workflow. Check out the Archives Unleashed team on Twitter and their website to find other ways to get involved and stay up to date with the work they’re doing.

"Why do we bring people together in one room to discuss #webarchives research?" asks @ianmilligan1 at #hackarchives. The answer: community, common vision, tool transparency, collective work. pic.twitter.com/rejr2eImKJ

— Justin Littman (@justin_littman) April 26, 2018

Many presentations, conversations, and tweets during the event indicated that working with web archives, particularly WARC files, can be messy, intimidating, and really difficult. The AUT tries to help simplify the process by breaking it down into four parts:

1. Filter - focus on a date range, a single domain, or specific content
2. Analyze - extract information that might be useful such as links, tags, named entities, etc.
3. Aggregate - summarize the analysis by counting, finding maximum values, averages, etc.
4. Visualize - create tables from the results or files for use in external applications, such as Gephi

#hackarchives The Archives Unleashed Toolkit cycle: filter-analyze-aggregate-visualize of WARCs https://t.co/ZvV6Hevjdvpic.twitter.com/qeenisOJeW

— Shawn M. Jones (@shawnmjones) April 26, 2018

We were encouraged to use the AUT throughout the event to go through the process of filtering, analyzing, aggregating, and visualizing for ourselves. Multiple datasets were provided to us and preloaded onto powerful virtual machines, provided by Compute Canada, in an effort to maximize the time spent working with the AUT instead of fiddling with settings and data transfers.

The datasets available to teams @ #hackarchives look phenomenal! How will they choose? Thanks to @ComputeCanada for the VMs, @ruebot + @ianmilligan for dataset prep & all of our dataset contributors! @nichworby@uoftlibraries@ubclibrary @coreyleedavis @justin_littman@walkerohpic.twitter.com/LHUYB8bUba

— The Archives Unleashed Project (@unleasharchives) April 26, 2018

Now that we knew the who, what, and why of the datathon, it was time to create our teams and get to work. We wrote down research questions (pink), datasets (green), and technologies/techniques (yellow) we were interested in using on sticky notes and posted them on a whiteboard. Teams started to form naturally from the discussion, but not very quickly, until we got a little help from Jimmy and Ian to keep things moving.

#hackarchives team formation with @lintool! Great topics and questions beginning to coalesce. pic.twitter.com/FYquBZUpDc

— Ian Milligan (@ianmilligan1) April 26, 2018

I worked with Jayanthy Chengan, Justin Littman, Shawn Walker, and Russell White. We wanted to use the #neveragain tweet dataset to see if we could filter out spam links and create a list of better quality seed URLs for archiving. Our main goal was to use the AUT without relying on other tools that we may have already been familiar with. Many of us had never even heard of Scala, the language that AUT is written in. We had all worked through the homework leading up to the datathon, but it still took us a few hours to get over the initial jitters and become productive.

Scala was a point of contention among many participants. Why not use Python or another language that more people are familiar with and can easily interface with existing tools? Jimmy had an answer ready, as he did for every question thrown at him over the course of the event.

My ?: For AUT, why Scala not Python or R? @lintools answer: Error reporting lost between JVM & Python. Scala more performant, “on the rails” for Spark. Using RDD instead of dataframes because didn’t exist when AUT written. Working on it: https://t.co/NersRff9Zy#hackarchives

— Justin Littman (@justin_littman) April 26, 2018

Around 5pm, it was time for dinner at Duke of York. My team decided against trying to get everyone up and running on their local machines, to enjoy dinner, and come back fresh for day 2.

#hackarchives Archives Unleashed after hours: the hacking takes a break, but the conversation continues. pic.twitter.com/RXI8YY9oDR

— Shawn M. Jones (@shawnmjones) April 26, 2018

In reality, it was more like:

• Jimmy Lin led a brainstorming session about moving the AUT from RDD to DataFrames. Samantha Fritz posted a summary of the feedback received where you can participate in the discussion.
• Nick Ruest talked about Warclight, a tool that helps with discovery within a WARC collection. He showed off a demo of it after giving us a little background information.
• Shawn Jones presented the five minute version of a blog post he wrote last year that talks about summarizing web archive collections.
• Justin Litmann presented TweetSets, a service that allows a user to derive their own Twitter dataset from existing ones. You can filter by any Tweet attributes such as text, hashtags, mentions, date created, etc.
• Shawn Walker talked about the idea of using something similar to a credit score to warn users, in realtime, of the risk that content they're viewing may be misinformation.

#hackArchives@ianmilligan1 starts the final session: each of the datathon teams is ready to present their work pic.twitter.com/PkZj6BX5Xo

— Shawn M. Jones (@shawnmjones) April 27, 2018

• As a Washington insider falls from grace, how quickly do those active in #MAGA and @realDonaldTrump shift allegiance?
• Did sentiment change towards Bannon before and after he was fired by Trump?

#hackArchives final projects! Here’s the first one, Make Tweets Great Again. Fascinating research questions which used sentiment scores with emojis and text-based sentiment analysis too. pic.twitter.com/oDsXrqveXi

— Ian Milligan (@ianmilligan1) April 27, 2018

Team Pipeline Research used datasets made up of WARC files of pipeline activism and Canadian political party pages, along with tweets (#NoASP, #NoDAPL,  #StopKM, #KinderMorgan). From the datasets, they were able to generate word clouds, find the image most frequently used, perform link analysis between pages, and analyze the frequency of hashtags used in the tweets. Through the analysis process, they discovered that some URLs had made it into the collection erroneously. 

#hackarchives Pipeline Research presents the data gathered from web archives with #AUT about different pipeline efforts: links, wordles, top images, hashtags pic.twitter.com/FNrH28KtQe

— Shawn M. Jones (@shawnmjones) April 27, 2018

#hackarchives The Spam Links team tries to answer "What is the best algorithm for extracting quality seed URLs from social media data?"pic.twitter.com/NgUKrfNu8M

— Shawn M. Jones (@shawnmjones) April 27, 2018

Team BC Teacher Labour was the final group of the day and they used a dataset from Archive-It about the British Columbia Teachers’ Labour Dispute. While exploring the dataset with the AUT, they created word clouds showing the frequency of words compared between multiple domains, network graphs showing named entities and how they related to each other, and many others. The most interesting visual they created was an animated GIF that quickly showed the primary image from each memento, giving a good overview of the types of images in the collection. 

#hackarchives topic analysis and visual discourse happening at the @archiveitorg collection at https://t.co/55Du1vkq20 extracted using #AUT: an animated gif of embedded images from each #memento gives a high level view of the collection pic.twitter.com/117pC0Qopd

— Shawn M. Jones (@shawnmjones) April 27, 2018

Team Just Kidding, There’s One More Thing was a team of one: Jimmy Lin. Jimmy was busy listening to feedback about Scala vs. Python and working on his own secret project. He created a new branch of the AUT running in a Python environment, enabling some of the things people were asking for at the beginning of Day 1. Awesome.

#hackarchives@lintool demonstrates what he has whipped up for #AUT in the past few days based on our feedback: PySpark with data frames working in a @ProjectJupyter notebook: now we will be able use #AUT within the #python ecosystem pic.twitter.com/nU550Qi9nU

— Shawn M. Jones (@shawnmjones) April 27, 2018

After Jimmy’s surprise, the organizers and teams voted for the winning project. All of the projects were great, but there can only be one winner and that was Team Make Tweets Great Again! I’m still convinced there’s a correlation between the number of emojis in their presentation, their team name, and the number of votes they received but 🤷🏻‍♂️. Just kidding 😂, your presentation was 🔥. Congratulations 🎊 to Shawn and his team! 

I’m brand new to the world of web archiving and this was my first time attending an event like this, so I had some trepidation leading up to the first day. However, I quickly discovered that the organizers and participants, regardless of skill level or background, were there to learn and willing to share their own knowledge. I would highly encourage anyone, especially if you’re in a situation similar to mine, to apply for the Vancouver datathon that was announced at the end of Day 2 or one of the 2019 datathons taking place in the United States.

We are pleased to announce the next #hackArchives datathon will be in #vancouver this November. We are so incredibly lucky to be working with our co-host @Reb_D and @SFU libraries. St pic.twitter.com/dBQOVk1eLi

— The Archives Unleashed Project (@unleasharchives) April 27, 2018

Thanks again to the organizers (Ian Milligan, Jimmy Lin, Nick Ruest, Samantha Fritz, Ryan Deschamps, and Nich Worby), their partners, and the University of Toronto for hosting us. Looking forward to the next one!

- Brian Griffin

#jcdl2018 stats. About 37% acceptance rate for full papers this year. Participation is increasing too. pic.twitter.com/H1vk3OB9KR

— Gianmaria Silvello (@giansilv) June 4, 2018

— Trevor Owens 💾🗄🕚 (@tjowens) June 4, 2018

Great crowd for my #jcdl2018 keynote! Room set up feels a little like looking out at some sort of intergalactic senate. pic.twitter.com/UTlrmyuVAo

— Trevor Owens 💾🗄🕚 (@tjowens) June 4, 2018

And @tjowens kicks off the #jcdl2018 opening keynote - appropriately titled “We Have Interesting Problems.” Really looking forward to this! pic.twitter.com/YWNAlHMjdf

— Ian Milligan (@ianmilligan1) June 4, 2018

@tjowens is talking about @benbendc's book in the #jcdl2018 keynote. pic.twitter.com/SHkRFKbK1N

— Mat Kelly (@machawk1) June 4, 2018

.@tjowens giving a nice run-down of cool funded IMLS projects - a nice shoutout to the WASAPI project. W/o WASAPI we’d be having a devil of a time with @unleasharchives - great community development. #jcdl2018

— Ian Milligan (@ianmilligan1) June 4, 2018

another @US_IMLS web archiving project: "Combining Social Media Storytelling with Web Archives"https://t.co/QLp3VP365m

supported @yasmina_anwar@acnwala@shawnmjones

some products:https://t.co/IcTJCYNzhPhttps://t.co/pulO7jNJt5
+ many papers#JCDL2018@tjowens@WebSciDL

— Michael L. Nelson (@phonedude_mln) June 4, 2018

Interested in how National Digital Platform and/or @librarycongress might help with distribution and discoverability of new forms of digital scholarship, esp SUP's Mellon-funded web-based digital publications. #jcdl2018https://t.co/zmenxltYbw

— Jasmine Mulliken (@jasminemulliken) June 4, 2018

Dr. Niall Gaffney emphasizing the importance of scientific reproducibility (keynote JCDL 2018)#jcdl2018pic.twitter.com/UKVoDrjwLg

— Alexander C. Nwala (@acnwala) June 5, 2018

Progress in Research vs. Reproducibility. Is it really tit for tat? #jcdl2018 keynote @de_Niledpic.twitter.com/ILxTpdqIXr

— Min-Yen Kan (@knmnyn) June 5, 2018

Texas Super Computing @jcdl2018#JCDL2018 presented by Niall Gaffney #stampede2pic.twitter.com/TrcdFiWINY

— Philipp Mayr (@Philipp_Mayr) June 5, 2018

. @de_Niled reminds me of @phonedude_mln‘s admonition to our @WebSciDL students: “No magic laptops!” #jcdl2018pic.twitter.com/5RmBS1wajL

— Michele Weigle (@weiglemc) June 5, 2018

#jcdl2018 session 4 has begun with @giansilv presenting "Evaluation of Conformance Checkers for Long-Term Preservation of Multimedia Documents". pic.twitter.com/tJWAMwZcLv

— Mat Kelly (@machawk1) June 5, 2018

Excited about @carlystrasser's keynote titled "Open Source Tech for Scholarly Communication: Why It Matters."#jcdl2018pic.twitter.com/mg695f2H5R

— Martin Klein (@mart1nkle1n) June 6, 2018

#jcdl2018@carlystrasser gives this morning's keynote: "Open Source Tech for Scholarly Communication: Why It Matters."pic.twitter.com/MosOOlgly5

— Shawn M. Jones (@shawnmjones) June 6, 2018

#jcdl2018@carlystrasser mentions the "chaos of the research process" and what is not necessarily being captured, with a focus on publications even though there are many other products of the scholarly process #scholarlycommunicationpic.twitter.com/V29h9P0HD7

— Shawn M. Jones (@shawnmjones) June 6, 2018

.@carlystrasser shared work of @olihb mapping scholarly collaboration https://t.co/YRFISJ6irnpic.twitter.com/88wyr30hlr

— Shawn M. Jones (@shawnmjones) June 6, 2018

#jcdl2018@carlystrasser shares Jennifer Lin's work of the "article nexus": https://t.co/FkEABBJCeQpic.twitter.com/eJI0fgrHfD

— Shawn M. Jones (@shawnmjones) June 6, 2018

.@carlystrasser gives a plug for @force11rescomm in October as well as other groups like @PLOS@ProjectJupyter@datadryad@biorxivpreprint and many others that are thinking about the future of research #jcdl2018pic.twitter.com/chSIIBWBZX

— Shawn M. Jones (@shawnmjones) June 6, 2018

#jcdl2018@carlystrasser lists #opensource software successes, best when collaborative and community-driven pic.twitter.com/wHJlQPECXf

— Shawn M. Jones (@shawnmjones) June 6, 2018

"Get everything into the browser" and "eliminate that production step in the middle" of #scholarlycommunication - @carlystrasser#jcdl2018pic.twitter.com/FZNbVBOSTL

— Shawn M. Jones (@shawnmjones) June 6, 2018

#jcdl2018@carlystrasser describes "the open ecosystem" including tools by @CokoFoundation and other tools like @inveniosoftwarepic.twitter.com/Ir5LBfR0hL

— Shawn M. Jones (@shawnmjones) June 6, 2018

#jcdl2018 isn't over yet, but #jcdl2019 will be @Illinois_Alma - @profdownie says that your best action plan is to plan to spend sometime in Chicago too! pic.twitter.com/m8ur0C42Xh

— Shawn M. Jones (@shawnmjones) June 6, 2018

After lunch photo Wednesday #jcdl2018pic.twitter.com/JPRmB1U0Wz

— Michele Whitehead (@WhiteheadML) June 6, 2018

-- Nwala (@acnwala)

On June 3, 2018, PhD students arrived in Fort Worth, Texas to attend the Joint Conference on Digital Libraries Doctoral Consortium. This is a pre-conference event associated with the ACM and IEEE-CS Joint Conference on Digital Libraries. This event gives PhD students a forum in which to discuss their dissertation work with others in the field. The Doctoral Consortium was well attended, not only by the presenting PhD students, their advisors/supervisors, and organizers, but also by those who were genuinely interested in emerging work. As usual, I live-tweeted the event to capture salient points. It was a very enjoyable experience for all.

Thanks very much to the chairs:

• J. Stephen Downie, University of Illinois at Urbana-Champaign, USA
• Oksana Zavalina, University of North Texas, USA
• Daniel Gelaw Alemneh, University of North Texas, USA
• Sampath Jayarathna, Old Dominion University, USA

In this post I will cover the work of all accepted students, three of whom are from the Web Science and Digital Libraries Research Group at Old Dominion University:

• Shawn M. Jones (me), Old Dominion University, USA
• Alexander Nwala, Old Dominion University, USA
• Mohamed Aturban, Old Dominion University, USA
• André Greiner-Petter, University of Konstanz, Germany
• Timothy Kanke, Florida State University, USA
• Hany Alsalmi, Florida State University, USA
• Corinna Breitinger, University of Konstanz, Germany
• Susanne Putze, University of Bremen, Germany
• Stephen Abrams, Queensland University of Technology, Australia
• Tirthankar Ghosal, Indian Institute of Technology Patna, India

I would also like to thank the assigned mentors of the Doctoral Consortium, who provided insight and guidance not only to their own assigned students, but the rest of us as well:

• Trond Aalberg, Norwegia University of Science and Technology, Norway
• Monika Akbar, University of Texas at El Paso, USA
• Daniel Alemneh, University of North Texas, USA
• Jose Borbinha, Técnico Lisboa, Portugal
• Sally Jo Cunningham, The University of Waikato, New Zealand
• Ying Ding, Indiana University Bloomington, USA
• Kai Eckert, Hochschule der Medien, Germany
• Daqing He, University of Pittsburgh, USA
• Martin Klein, Los Alamos National Laboratory, USA
• Philipp Mayr-Schlegel, GESIS – Leibniz Institute for the Social Sciences, Germany
• Mirella M. Moro, Universidade Federal de Minas Gerais, Brazil
• Federico Nanni, University of Mannheim, Germany
• Michele C. Weigle, Old Dominion University, USA
• Jian Wu, Pennsylvania State University, USA

WS-DL Presentations

Shawn M. Jones

.@shawnmjones presenting "improving collection understanding in web archives" #jcdl2018@WebSciDL@jcdl2018pic.twitter.com/vq3mJrMSen

— Michael L. Nelson (@phonedude_mln) June 3, 2018

How does a researcher differentiate between web archive collections that cover the same topic? Some web archive collections consist of 100,000+ seeds, each with multiple mementos. There are more than 8000 collections in Archive-It as of the end of 2016. Existing metadata in Archive-It collections is insufficient because the metadata is produced by different curators from different organizations applying different content standards and different rules of interpretation. As part of my doctoral consortium submission, I proposed improving upon the solution piloted by Yasmin AlNoamany. She generated a series of representative mementos and then submitted them to the social media storytelling platform Storify in order to provide a summary of each collection.

Improving Collection Understanding in Web Archives by @shawnmjones@WebSciDL at #JCDL2018#JCDLDC2018#DoctoralConsortiumpic.twitter.com/AKzNgNZWpA

— Sawood Alam (@ibnesayeed) June 3, 2018

As part of my preliminary work I presented some findings that will be published at iPres 2018. We discovered four semantic categories of Archive-It collections: collections where an organization archived itself, collections about a specific subject, collections about expected events or time periods, and collections about spontaneous events. The collections AlNoamany used in her work fit into the last category. This also turned out to be the smallest category of collections, meaning that there are many other types of collections not evaluated by her method. She proved that humans could not tell the difference between her automatically-generated stories and other stories generated by humans. She did not, however, provide evidence that the visualization was useful for collection understanding. We also have the problem that Storify is no longer in service, something that I mentioned in a previous blog post. My plan includes developing a flexible framework that allows us to test different methods of selecting representative mementos. This framework will also allow us to test different types of visualizations using those representative mementos. Some of these visualizations may make use of different social media platforms. I plan to evaluate these collections by first creating user tasks that give us some idea that a user understands aspects of a collection. With these tasks I intend to then evaluate different solutions via user testing. The solutions that score best from the testing will address a large problem inherent to the scale of web archives.

In analyzing @archiveitorg collections, @shawnmjones found 54% of the collections are people/orgs archiving themselves, 24% archiving a subject, 14% archiving event they know is coming, and 4% collections are unexpected events. #jcdl2018

— Mat Kelly (@machawk1) June 3, 2018

Alexander Nwala

How do we find high quality seeds for generating web archive collections? Alexander is focusing on a different aspect of web archive collections than I am. I am analyzing existing collections. He is building collections from seeds supplied by social media users. He notes that users often create "micro-collections" of web resources, typically surrounding an event. Using examples like ebola epidemics, the Ukraine crisis, and school shootings, Alexander asks if seeds generated by social media are comparable to those generated by professional curators. He also seeks quantitative methods for evaluating collections. Finally, he wants to evaluate the quality of collections at scale.

.@acnwala - "not all micro-collections in social media will give us high quality seeds"- how do we handle this at scale? @jcdl2018#jcdl2018#DoctoralConsortiumpic.twitter.com/Tk2B3bthZz

— Shawn M. Jones (@shawnmjones) June 3, 2018

He demonstrated the results of using a prototype system that extracts seeds from social media and compared these seeds to those extracted from Google search engine result pages (SERPs). He discovered that, when using SERPs, the probability of finding a URI for a news story diminishes with time. He introduced methods like distribution of topics, distribution of sources, distribution in time, content diversity, collection exposure, target audience, and more. He covered some of his work on the Local Memory Project as well as work that will be presented at JCDL 2018 and Hypertext 2018. He intends to do further research on hubs and authorities in social media, as well as evaluating the quality of collections. Alexander will ensure that good quality seeds make it into web archives, addressing an aspect of curation that has long been an area of concern in web archives.

re: @acnwala's use of scraping Google UI vs. bing API, APIs and UIs can give very different results, see: "Agreeing to Disagree: Search Engines and their Public Interfaces" from #JCDL2007, @fmccownhttps://t.co/BnGSTzqS91
#jcdl2018

— Michael L. Nelson (@phonedude_mln) June 3, 2018

Mohamed Aturban

How can we verify the content of web archives? Mohamed presented his work on fixity for mementos. He described issues with temporal violations and playback issues. He asked whether different web archives agreed on the content of mementos produced for the same live resource at the same time. He showed how "evil" archives could potentially manipulate memento content to produce a different page than existed at the time of capture. So, how do we ensure that the memento was unaltered since the time of capture?

.@maturban1 - existing hashing functions are insufficient for generating fixity of mementos in a #webarchive - we need a Memento-aware fixity solution, including all embedded resources #jcdl2018@jcdl2018#DoctoralConsortiumpic.twitter.com/5yMkn9MPJx

— Shawn M. Jones (@shawnmjones) June 3, 2018

He demonstrated that the playback engine used by a web archive can inadvertently change the result of the displayed memento. Just providing a timestamped hash of the memento HTML is not enough. He proposes generating a cryptographic hash for the memento and all embedded resources and then generating a manifest of these hashes. This manifest will then be stored as mementos themselves in multiple web archives. I expect this work to be quite important to the archiving community, addressing a concern that many professional archivists have had for quite some time.

.@maturban1 - once we have fixity information for mementos, we will push a manifest with this information into multiple web archives #jcdl2018@jcdl2018#DoctoralConsortiumpic.twitter.com/Dqerbxrb26

— Shawn M. Jones (@shawnmjones) June 3, 2018

Other Work Presented

André Greiner-Petter

Here's @GreinerPetter's suggestion for a semantification helper for mathematical formulae in Wikipedia for example. Nice! #jcdl2018pic.twitter.com/Vk8kIozVAS

— Mandy (@protestreich) June 3, 2018

Research papers use equations all of the time. Unfortunately, there isn't a good method of comparing equations or providing semantic information about them. André Greiner-Petter is working on creating a method of enriching the equations used in research papers. This will have a variety of uses, such as detecting plagiarism or finding related literature.

Timothy Kanke

Timothy Kanke from @floridastate presents different concepts from @wikidata: items, projects, WikiProjects https://t.co/jPcXO16dKA#jcdl2018#DoctoralConsortium@jcdl2018pic.twitter.com/0T7meZJ6ht

— Shawn M. Jones (@shawnmjones) June 3, 2018

How are people using Wikidata? I had attended a session on Wikidata at Wiki ConferenceUSA 2014, but have not really examined it since. Will it be useful for me? How do I participate? Who is involved? Timothy Kanke seeks to understand the answers to all of these questions. The Wikidata project has grown over the last few years, feeding information back into the Wikipedia community. Kanke will study the Wikidata community and provide a good overview for those who want to use its content. Using his work, we will all have an understanding of the overall ways in which Wikidata can work for the scholarly community.

Hany Alsalmi

"Dual Language Information Seeking in Digital Libraries" talk by @HanyAlsalmi in #JCDL2018#DoctroalConsortium for #Arabic is very relatable to many other languages like #Persian and #Urdu. pic.twitter.com/dnRXsgLFi6

— Sawood Alam (@ibnesayeed) June 3, 2018

How many languages do you use for searching? What is the precision of the results when you switch languages, even for the same query? Hany Alsalmi noticed that users who search in English were getting different results than when they searched for the same term in Arabic. Alsalmi will perform studies on users of the Saudi Digital Library to understand how they perform their searches and how successful those searches are. He will also record their reactions to search results, with the concern being that the user will quit in frustration if the results are insufficient. His work will have implications for search engines in the Arabic-speaking world.

Corinna Breitinger

.@BreitingerC mentions different types of semantic similarity measures for use in analyzing scholarly literature: I was familiar with text-based measures and citation-based measures, but approaches for analyzing mathematical language and figures are new to me #jcdl2018pic.twitter.com/U8prxk5hUO

— Shawn M. Jones (@shawnmjones) June 3, 2018

Scholarly recommendation systems examine papers using text similarity. Can we do better? What about the figures, citations, and equations? Corinna Breitinger will take all of these text-independent semantic markers into consideration with the development of a new recommender approach targeted at STEM fields. Once that is done, she will create a new visualization concept that will help users view and navigate a collection of similar literature. The benefits of such a system will help spot redundant research and also help us find related research in the field.

Susanne Putze

Suzanne Putze presents "Digital Libraries for Experimental Data"@jcdl2018#DoctroalConsortium#jcdl2018pic.twitter.com/m1KDSj5EmB

— Shawn M. Jones (@shawnmjones) June 3, 2018

How is research data managed? How can we facilitate making data management a “first-class citizen”? To do so would improve the amount of data shared by researchers as well as its quality. Susanne Putze has extended experiment models to improve data documentation. She will create prototypes and evaluate how well they work to address data management in the scholarly process. From there she will begin the process of improving knowledge discovery using these prototypes. Her research has implications for how we handle our data and incorporate it into scholarly communications.

Stephen Abrams

I was composing my tweet on this same thing! :-)

"In science, truth is not binary, instead we approach to it asymptotically!" -- Stephen Abrams#JCDL2018#JCDLDC2018#DoctoralConsortium

— Sawood Alam (@ibnesayeed) June 3, 2018

How successful are digital preservation efforts? Stephen Abrams is working on creating metrics for this purpose. He is planning on evaluating digital preservation from the perspective of communications rather than through preservation management concepts like quantities, ages, of quality of preserved material. Thanks to his presentation I will now examine terms like “verisimilitude”, “semiotic”, and “truthlikeness”. When he is done, we should have better metrics to evaluate things like the trustworthiness of preserved material. His work is more general and theoretical than Mohamed’s, but there is a loose connection to be sure.

Tirthankar Ghosal

Tirthankar Ghosal's focus is helping editors to flag editors and papers ahead of time to determine the potential for a paper to be accepted or rejected - @weiglemc mentions that determining novelty of research is a "huge task"@jcdl2018#jcdl2018#DoctoralConsortiumpic.twitter.com/gGDW8vyC5b

— Shawn M. Jones (@shawnmjones) June 3, 2018

Why are papers rejected by editors? Have we done a good job identifying what makes our paper novel? What if we could spot such complex issues in our papers prior to submission? Tirthankar Ghosal seeks to help address these concerns by using AI techniques to help researchers and editors more easily identify papers that will likely be rejected. He has already done some work examining reasons for desk rejections. He will identify methods for detecting what makes a paper novel, if a paper is fit for a given journal, if it is of sufficient quality to be accepted, and lastly create benchmark data that can be used to evaluate papers in the future. His work has large implications for scholarly communication and may affect not only the way we write, but also how submissions are handled in the future.

What Next?

#jcdl2018@profdownie mentions to #DoctoralConsortium participants @jcdl2018: "a successful thesis is really focused", "really focused on the limitations", note what you have not explored but may explore in the future

— Shawn M. Jones (@shawnmjones) June 3, 2018

#jcdl2018#doctoralconsortium@profdownie asks Stephen Abrams What is the "impact on the world" of your thesis? "What is this thing in the future that we want better of?" Things to think of when considering my own work...

— Shawn M. Jones (@shawnmjones) June 3, 2018

I would like to thank all participants for their input and insight throughout the event. Hearing their feedback for other participants was quite informative to me as well. We will all have improved candidacy proposals as a result of their input and, more importantly, will use this input to improve our contributions to the world.

#jcdl2018 Close the doctoral consortium by a nice photo~ pic.twitter.com/vTUTN7AEuT

— Chenwei Zhang (@z_vvvv) June 3, 2018

Updated on 2018/06/09 at 20:50 EDT with embed of Mohamed Aturban's Slideshare.

--Shawn M. Jones

Mat Kelly reports on the Web Archiving and Digital Libraries (WADL) Workshop 2018 that occurred in Fort Worth, Texas.                                                                                                                                                                                                                                                                                                                                                                            ⓖⓞⓖⓐⓣⓞⓡⓢ

On June 6, 2018, after attending JCDL 2018 (trip report), WS-DL members attended the Web Archiving and Digital Libraries 2018 Workshop (#wadl2018) in Fort Worth, Texas (see trip reports from WADL 2017, 2016, 2015, 2013). WS-DL's contributions to the workshop included multiple presentations inclusive of the workshop keynote by my PhD advisor, which I discuss below.

.@mart1nkle1n getting us started at #WADL2018#jcdl2018https://t.co/SctiuI9foXpic.twitter.com/Kbj4ofRwRH

— Michael L. Nelson (@phonedude_mln) June 6, 2018

The Project Panel

Martin Klein (@mart1nkle1n) initially welcomed the workshop attendees and had the group of 26-or-so participants give a quick overview of who they were and their interest in attending. He then introduced Zhiwu Xie (@zxie) of Virginia Tech to begin the series of presentations reporting on the project kickoff of the IMLS-funded project (as establish at WADL 2017) "Continuing Education to Advance Web Archiving". A distinguishing feature of this project compared to others, Zhiwu said, is that the projects will use project-based problem solving instead of the products being surveys and lectures. He highlighted a collection of Curriculum modules involving existing practice (event archiving) to feed into various Web archiving tools (e.g., Social Feed Manager (SFM), ArchiveSpark, and Archives Unleashed Toolkit) to facilitate the understanding of the fundamentals (e.g., web, data science, big data) to produce experience in libraries, archives, and programming. The focus here was on individuals that had some level of prior experience with archives instead of the program being designed as training for those with zero experience in the area.

Now @zxie kicks off our IMLS project - “Continuing Education to Advance Web Archiving.” Funded under the Laura Bush 21st Century Librarian Program. We’ll be developing some #webarchiving training. #wadl2018#jcdl2018pic.twitter.com/9ihZejeUrm

— Ian Milligan (@ianmilligan1) June 6, 2018

@phonedude describing some approaches that may be used for storytelling with archives now that @storify is gone. #wadl2018pic.twitter.com/E8joxzeoOF

— Mat Kelly (@machawk1) June 6, 2018

ODU WS-DL's Michael Nelson (@phonedude_mln) continued with the one motivation is to encourage storytelling using Web archives and how that has been hampered with the recent closing of Storify. Some recent work of the group (including the in-develop project MementoEmbed) would allow this concept to be revitalized despite Storify's demise through systematic "card" generation of mementos to allow a more persistent (in the preservation sense) version of the story to be extracted and retained.

@justin_littman starts his #wadl2018 talk in distinguishing what you get from the Twitter API compared to what you would see on their Web interface. He then highlights some tools to interact with the data then describes doing so with their @US_IMLS-supported Social Feed Manager. pic.twitter.com/8Y1uYSMmta

— Mat Kelly (@machawk1) June 6, 2018

Justin Littman (@justin_littman) of George Washington University Libraries continued the project description by describing Social Feed Manager's and emphasized that what you get from the Twitter API may well differ from what you get from the Web interface. The purpose of SFM is to be an easy-to-use, self-service Web interface to drive down the barriers in collecting social media data for academic research.

At #jcdl2018#wadl2018 gave a quick run down of our @unleasharchives projects – for some more info as promised on the research approach to using our tools, here’s the “FAAV” cycle. pic.twitter.com/68gxWLn0eG

— Ian Milligan (@ianmilligan1) June 6, 2018

Ian Milligan (@ianmilligan1) continued by giving a quick run-down of his group's Archives Unleashed Projects, noting a realization in the project's development that not all historians like working with the command-line and Scala. He then briefly described the projects' approach of a filter-analyze-aggregate-visualize to make using large collections of Web archives more effective for research.

Edward A. Fox presents work done archiving events @virginia_tech#WADL2018#jcdl2018https://t.co/XfbWQILkDMpic.twitter.com/CY18bk9Cbv

— Shawn M. Jones (@shawnmjones) June 6, 2018

Wrapping up the project report, Ed Fox described Virginia Tech's initial attempts at performing crawls with Heritrix via Archive-It and how noisy the results were. He emphasized that a typical crawling approach consisting of starting with seed URIs harvested from tweets does not work well. The event model his group is developing and further evaluating will help guide the crawling procedure.

Ed's presentation completed the series of reports for the IMLS project panel and began a series of individuals presenting.

#jcdl2018#wadl2018@johnaberlin presenting: "Swimming In A Sea Of JavaScript, Or: How I Learned To Stop Worrying And Love High-Fidelity Replay"

Based on his @WebSciDL MS thesis https://t.co/xeAlv0pVVFpic.twitter.com/FDcxodBuFu

— Michael L. Nelson (@phonedude_mln) June 6, 2018

Individual Presentations

John Berlin (@johnaberlin) started off with an abbreviated version of his Master's Thesis titled, "Swimming In A Sea Of JavaScript, Or: How I Learned To Stop Worrying And Love High-Fidelity Replay". While John had recently given his defense in April (see his post for more details), this presentation focused on some of the more problematic aspects of archival replay as caused by JavaScript. He highlighted specific instances where the efforts of a replay system to accurately replay JavaScript varied from causing a page to display a completely blank viewport (see CNN.com has been unarchivable since November 1st, 2016) to the representation being highjacked to declare Brian Williams as the originator of "Gin and Juice" long before Snoop Dogg(y Dogg). John has created a Chrome and Firefox extension he dubbed "Wayback Plus Plus" that mitigates JavaScript-based replay issues using client-side redirects. See his presentation for more details.

#wadl2018@edwardafox on URLs in tweets - time interval between a tweet and its webpage - most people prefer newly released web pages - for successfully retrieved URLs 50% were archived within 5 days after tweets were posted #jcdl2018pic.twitter.com/aQqPIoBr13

— Shawn M. Jones (@shawnmjones) June 6, 2018

The workshop participants then had a break to grab a boxed lunch and followed with Ed Fox, again, presenting "A Study of Historical Short URLs in Event Collections of Tweets". In this work Ed highlighted the number of tweets in their collections that had URLs, namely that 10% had 2 URLs and less than 0.5% had 3 or more. From this collection, his group analyzed how many of the URLs linked are still accessible in Internet Archive's Wayback Machine with an emphasis that the Wayback Machine is not covering a lot of things that are in the Twitter data he has gathered. His group also analyzed the time difference between when a tweet with URLs was made and when it was archived and found that 50% were archived within 5 days after the tweet was posted.

Keynote

The workshop keynote, "Enabling Personal Use of Web Archives" was next and presented by my PhD Advisor Dr. Michele C. Weigle (@weiglemc). Her presentation initially gave a high-level overview of the needs of those that want to perform personal Web archiving and the tools that the WS-DL group have created over the years in facilitating the efforts to address those needs. She highlighted the early work of the group in identifying disasters in existing archives with a segue of the realization that many archive users lack in that there are more archives beyond Internet Archive.

#wadl2018@weiglemc mentions that there are multiple #webarchives - with ArchiveNow we can submit to many web archives at once https://t.co/uAQqOenQOe#jcdl2018pic.twitter.com/1KqMd6b9yh

— Shawn M. Jones (@shawnmjones) June 6, 2018

In her (our) group's tooling to encourage Web users to Archive What They See Now, they created the WARCreate Chrome extension to create WARC files from any Web page. To resolve the issue of what a user is to do with their WARCs, they then created the Web Archiving Integration Layer (WAIL) (and later an Electron version) to allow individuals to control both the preservation and replay process. To give users a better picture of the archived Web as they browsed, they created the Chrome extension Mink to give users a measure of how well-archived (in terms of quantity) a URI is as they browsed the live Web and optionally (and easily) submit the URI currently viewed to 1-to-3 Web archives.

There are a lot of tools available at the @WebSciDL Group, our GitHub page has a lot of projects in progress: https://t.co/9qTndi5Vxd#wadl2018#jcdl2018pic.twitter.com/LZQFjaongt

— Shawn M. Jones (@shawnmjones) June 6, 2018

Dr. Weigle also highlighted the work of other WS-DL students of past and present like Yasmin Anwar's (@yasmina_anwar) Dark and Stormy Archives (DSA) and Shawn Jones' (@shawnmjones) upcoming MementoEmbed tool.

Following a tool review, Dr. Weigle asked, "What if browsers could natively interpret and replay WARCs?". She performed a high level review of what could be possible if the compatibility barriers between the archived and live Web were resolved through live Web tools that could natively interact with the archived Web. In one example, she provided a screenshot where in-place of the "secure" badge a browser provides, it might also be aware that it is viewing an archived page and indicate as such.

.@weiglemc: What if browsers could natively interpret and replay WARC files?
me: My life would be complete, that's what. Yes! #wadl2018#JCDL2018

— Jasmine Mulliken (@jasminemulliken) June 6, 2018

Libby Hemphill (@libbyh) presented next with "Developing a Social Media Archive at ICPSR" where her group sought to make data useful for people who wanted to understand how we are today from the perspective of people of the long-distant future. She mentioned how messy it can be to consider the ethical challenges when archiving social media data and that people have different levels of comfort depending of what sort of research for which their social media content is to be used. She outlined an architecture of their social media archive SOMAR for federating data to follow the terms of service, rehydrating tweets to follow the terms of research, and other aspects of the social-media-to-research-data process.

Which looks roughly similar to the SOMAR infrastructure described by @libbyh. #wadl2018pic.twitter.com/l7aViIBf8R

— Justin Littman (@justin_littman) June 6, 2018

The workshop then took another break with a simultaneous poster session including a poster by Justin Littman titled, "Supporting social media research at scale" and WS-DL's Sawood Alam's (@ibnesayeed) "A Survey of Archival Replay Banners". Just prior to their poster presentations, each gave a lightning talk as a quick overview to entice attendees into stopping by.

#jcdl2018@justin_littman is doing a lightning talk for his "Supporting social media research at scale" poster at #wadl2018pic.twitter.com/P03L2GRsuU

— Shawn M. Jones (@shawnmjones) June 6, 2018

.@ibnesayeed is presenting his #wadl2018 poster "A Survey of Archival Replay Banners" detailing issues with #webarchive banners, such as covering navigational links on a memento, and more... #jcdl2018pic.twitter.com/OGWNWHyo5h

— Shawn M. Jones (@shawnmjones) June 6, 2018

An excellent use of a Merkle Tree (https://t.co/lngldKeXwn) by @maturban1 to generate a root hash for each memento that includes the hashes of each of its embedded resources #jcdl2018#wadl2018pic.twitter.com/azY4XdRIIj

— Shawn M. Jones (@shawnmjones) June 6, 2018

After the break, WS-DL's Mohamed Aturban (@maturban1) presented "It is Hard to Compute Fixity on Archived Web Pages". Mohamed's work highlighted an issue that subtle changes in content may be difficult to detect using conventional hashing methods to compute the fixity of Web pages. He emphasized that computing the fixity of the root HTML page of a memento is not enough for fixity and that the fixity must also be computed for all embedded resources. With an approach utilizing Merkle trees (or on WP), he generates a hash of the composite memento representative of the fixity of all embedded resources. In one example highlighted in his recent post and tech report, Mohamed showed the manipulation of Climate Change data.

To wrap up the presentations for the workshop, I (Mat Kelly, @machawk1) presented "Client-Assisted Memento Aggregation Using the Prefer Header". This work highlighted one particular aspect of my presentation the previous day at JCDL 2018 (see blog post), namely of how the framework in the basis presentation facilitates the specification of which archives are aggregated using Memento. The previous investigation by Jones, Van de Sompel et al. (see "Mementos in the Raw, Take Two") used the HTTP Prefer header to allow a client to request the un-rewritten version of mementos from archival replay system. In my work, I imagined a more capable Memento aggregator that would expose the archives aggregated and allow a client, basing their customizations off of the aggregator's response, customize the set of archives aggregated by sending the set as base64-encoded data in the Prefer request header.

#jcdl2018#wadl2018@machawk1 is presenting "Client-Assisted Memento Aggregation Using the Prefer Header" as part of his work on aggregating memento data from multiple web archives pic.twitter.com/JmhTHdKCcA

— Shawn M. Jones (@shawnmjones) June 6, 2018

Closing

When I was through with the final presentation, Ed Fox began the wrap-up of the conference. This discussion of all attendees opened the floor for comments and recommendations for the future of the workshop. With the discussion finished, the workshop came to a close. As usual, I found this workshop extremely informative, though I was familiar with many of the participants previous work. I am hoping, as also expressed by other attendees, to encourage other fields to become involved and present their ongoing work and ideas at this informal workshop. Doing so, from the perspective of both an attendee and presenter, has proven valuable.

—Mat (@machawk1)

Fort Worth Museum of Science & History 9/11 Tribute

The theme of the workshop on Knowledge Discovery from Digital Libraries (KDDL) was to uncover hidden relationships between data with techniques from artificial intelligence, mathematics, statistics, and algorithms. The workshop organizers, which included ODU Computer Science alumna, Dr. Hui Shi, Dr. Wu He, and Dr. Guandong Xu identified the following objectives that we were to explore:

• Existing and novel techniques to extract and present knowledge from digital libraries;
• Advanced ways to organize and maintain digital libraries to facilitate knowledge discovery;
• Knowledge discovery applications in business; and
• New challenges and technologies brought to the area of knowledge discovery and digital libraries.

Hui Shi, @oducs alumni, kicks off Knowledge Discovery from Digital Libraries (KDDL) 2018 #jcdl2018pic.twitter.com/NrLJUlfRm0

— Shawn M. Jones (@shawnmjones) June 6, 2018

The KDDL workshop consisted of three paper presentations which are summarized here.

Presentation 1: I presented my work on Mining the Web to Approximate University Rankings based on the tech report "University Twitter Engagement: Using Twitter Followers to Rank Universities" (https://arxiv.org/abs/1708.05790) and discussed in an earlier blog post.


This paper presented an alternative methodology for approximating the academic rankings of a university using social media; specifically, the university's Twitter followers. We identified a strategy for discovering official Twitter accounts along with a comparative analysis of metrics mined from the web which could be predictors of high academic rank (e.g., athletic expenditures, undergraduate enrollment, endowment value). As expected, schools with more financial resources tend to have more Twitter (@Twitter) followers based on larger enrollments, big endowments, and big investments in their sport programs. We also discovered that smaller schools like Wake Forest University can enhance their reputation when they employ faculty with national name recognition (e.g., Melissa Harris-Perry (@MHarris-Perry)).  For those wishing to perform further analysis, we have posted all of the ranking and supporting data used in this study which includes a social media rich data set containing over 1 million Twitter profiles, ranking data, and other institutional demographics in the oduwsdl Github repository.

Knowledge Discovery from Digital Libraries (KDDL) first session is kicked off by @WebSciDL member @CorrenMcCoy presenting "Mining the Web to Approximate University Rankings" available at https://t.co/hILDimq46I#jcdl2018pic.twitter.com/cbhTe8Iu89

— Shawn M. Jones (@shawnmjones) June 6, 2018

Presentation 2: Basic Science and Technological Innovation: A Classification of Research Publications was presented by Dr. Robert M. Patton, Oak Ridge National Laboratory. This paper explored the context required for funding decision makers, sponsors, and the general public to determine the value of research publications. Core questions addressed the accessibility of massive digital libraries and methods related to identification of new discoveries, data sets, publications in disparate journals, and new software codes. Dr. Patton asserted that research evaluation has become increasingly complicated and citation analysis alone is insufficient if considered within the context of the people who control the flow of funding. His presentation of evaluation techniques included altmetrics along with a comparison of Bohr’s, Edison’s, and Pasteur’s quadrants as classifiers which use the wording of titles and abstracts in conjunction with domain specific terminology.

A Classification of Research Publications

Presentation 3: Introducing Math QA -- A Math Aware Question Answering System was presented by Felix Hamborg, University of Konstanz. This paper presented a software tool that allows a user to enter a textual request for a math formula (e.g., What is the formula for …?) in English or Hindi and is then presented with the required parameters and the actual formula from Wikidata. The authors mined 40 million articles in Wikidata searching for <math> tags to identify 17 thousand general and geometric formulas. They defined a QA System workflow consisting of three distinct modules for calculation, question parsing, and formula retrieval. Their discovery of geometric formulas (e.g., polygons, curves) was slightly more complex as these formulas can include a nested hierarchy of related data that required traversal of the associated Wikidata subsections. Following evaluation and comparison to a commercial engine, exported information was parsed and ported back into Wikidata. The author's source code and data is available in their GitHub repository (http://github.com/ag-gipp/MathQa).

A Math Aware Question Answering System

Following the paper presentations, the workshop participants divided into two groups to conduct a breakout session where we discussed Challenges and Research Trends in Knowledge Discovery from Digital Libraries and Beyond.  Each group was asked to offer opinions and provide summary responses for each of the following topics:

• What are your reactions to the paper presentations? What did you learn that you didn’t previously know?
• What are the current techniques, applications, and/or research questions that you are addressing in Knowledge Discovery from Digital Libraries and Beyond? What are the biggest impediments or challenges limiting Knowledge Discovery from Digital Libraries and Beyond?
• What are your top priorities in implementing Knowledge Discovery from Digital Libraries and Beyond? 
• What resources and/or support do you need to implement? 
• What areas will you recommend for research? How do you think artificial intelligence (AI) can benefit knowledge discovery in digital libraries? 
• Suggestions for coordination of research and future collaboration.

Collectively, my group's responses centered on the themes of data curation with less reliance on subject matter experts, methods or tools to make data more self-documenting, and new strategies for relationship extraction between linked entities. There was also considerable discussion related to reproducible research using common repositories and formats conducive to sharing data (e.g., XML) and open access to both software and the peer review process.

I would like to thank Old Dominion University for the Graduate Student Travel Award which helped to facilitate my participation in the JCDL conference and this workshop.

--Corren (@correnmccoy)

This may sound familiar, but yet again I'm way behind in posting about my previous offerings of CS 725/825 Information Visualization.
(Previous semester highlights posts: Spring 2016, Spring 2015, Spring/Fall 2013, Fall 2012, Fall 2011)

Here are a few projects that I'd like to highlight from Spring 2017. (All class projects are listed in my InfoVis Gallery.  This semester has its own page because there were 19(!) projects.)  All of the projects were implemented using the D3.js library.

Because the Spring 2017 semester began with President Donald Trump's Travel Ban (EO 13769) and we have a large international graduate student population, students were understandably interested in US immigration and refugee data.  The first two projects here focus on that.  In addition, one project looked at sentiment about the US Presidential candidates on social media on Election Day.

The last two projects that I'll highlight are focused on the lighter topic of sports, NFL football and IPL cricket.



This project focuses on refugee admittance to the US between 2008-2016. The visualization highlights the number of refugees by country of nationality/religion, and relationship between a country's number of refugees, number of war deaths, population, GDP per capita (GDPPC), and State Fragility Index (SFI).

Inspired by AlNoamany's work from "Detecting off-topic pages within TimeMaps in Web archives" I am pleased to announce an alpha release of the Off-Topic Memento Toolkit (OTMT). The results of testing with this software will be presented at iPres 2018 and those results are now available as a preprint.

Web archive collections are created with a specific purpose in mind. A curator will supply seeds for the collection and create multiple versions of these seeds in order to study the evolution of a web page over time. This is valuable for following the changes in an organization or the events in a news story. Unfortunately, depending on the curator's intent, sometimes these seeds go off-topic. Because web archive crawling software has no way to know that a page is off-topic, these mementos are added to the collection. Below I list a few examples of off-topic pages within Archive-It collections.

This memento from the Human Rights collection at Archive-It created by the Columbia University Libraries is off-topic. The page ceased to be available at some point and produced this "404 Page Not Found" response with a 200 HTTP status.

This memento from the Egypt Revolution and Politics collection at Archive-It created by the American University in Cairo is off-topic. The web site began having database problems.

It is important to note that the OTMT does not delete potentially off-topic mementos, but rather only flags them for curator review. Detecting such mementos allows us to exclude them from consideration or flag them for deletion by some downstream tool, which is important to our collection summarization and storytelling efforts. The OTMT detects these mementos using a variety of different similarity measures. One could also use the OTMT to detect and study off-topic mementos.

Installing the software

The OTMT requires Python 3.6. Once you have met that requirement, install OTMT by typing:

# pip install otmt

This installs the necessary libraries and provides the system with a new detect-off-topic command.

A simple run

To perform an off-topic run with the software on Archive-It collection 1068, type:

# detect-off-topic -i archiveit=1068 -tm cosine,bytecount -o myoutputfile.json

This will find all URI-Rs (seeds) related to Archive-It collection 1068, download their timemaps (URI-Ts), download the mementos within each timemap, process those mementos via the default similarity measures, and write the results in JSON format out to a file named outputfile.json.

The JSON output looks like the following.

"http://wayback.archive-it.org/1068/timemap/link/http://www.badil.org/": { "http://wayback.archive-it.org/1068/20130307084848/http://www.badil.org/": { "timemap measures": { "cosine": { "stemmed": true, "tokenized": true, "removed boilerplate": true, "comparison score": 0.10969941307631487, "topic status": "off-topic" }, "bytecount": { "stemmed": false, "tokenized": false, "removed boilerplate": false, "comparison score": 0.15971409055425445, "topic status": "on-topic" } }, "overall topic status": "off-topic" }, ...

Each URI-T serves as a key containing all URI-Ms within that timemap. In this example the timemap at URI-T http://wayback.archive-it.org/1068/timemap/link/http://www.badil.org/ contains several mementos. For brevity, we are only showing results for the memento at http://wayback.archive-it.org/1068/20130307084848/http://www.badil.org/.

The key "timemap measures" contains all measures run against the memento. In this case I used the two measures "cosine" and "bytecount". Each measure entry indicates which preprocessing has been performed against that memento (e.g., stemmed, tokenized, and removed boilerplate). Under "comparison score" is that measure's score. Under "topic status" is a verdict on whether or not the memento is on or off-topic. Finally, the "overall topic status" indicates if any of the measures determined that the memento is off-topic.

The OTMT uses an input-measure-output architecture. This way the tool separates the concerns of input, (e.g., how to process Archive-It collection 1068 for mementos), from measure (e.g., how to process these mementos using cosine and byte count similarity measures), and output (e.g., how to produce the output in JSON format and write it to the file outputfile.json). This architecture is extensible, providing interfaces allowing for more input types, measures, and output types to be added in the future.

The -i (for specifying the input) and -o (for specifying the output) options are the only required options. The following sections detail the different command line options available to this tool.

Input and Output

The input type is supplied by the -i option. OTMT currently supports the following input types:

• an Archive-It collection ID (keyword: archiveit)
• one or more TimeMap URIs (URI-T) (keyword: timemap)
• one or more WARCs (keyword: warc)

An output file is supplied by the -o option. Output types are specified by the -ot option. OTMT currently supports the following output types:

• JSON as shown above (the default) (keyword: json)
• a comma-separated file consisting of the same content found in the JSON file (keyword: csv)

To specify multiple WARCs, list them after the warc option like so:

# detect-off-topic -i warc=mycrawl1.warc.gz,mycrawl2.warc.gz -o myoutputfile.json

Likewise, for multiple TimeMaps, list them with the timemap argument and separate their URI-Ts with commas, like so:

# detect-off-topic -i timemap=https://archive.example.org/urit/http://example.org,https://archive.example.org/urit/http://example2.org -o myoutputfile.json

To use the comma-separated file format instead of json use the -ot option as follows:

# detect-off-topic -i archiveit=3936 -o myoutputfile.csv -ot csv

For better processing, we want to eliminate any interference from HTML and JavaScript associated with archive-specific branding. In the case of TimeMaps and Archive-It collections, raw mementos will be downloaded where available. While any TimeMap may be specified for processing, raw mementos are preferred as they do not contain the additional banner information and other augmentations supplied by many web archives. These augmentations may skew the off-topic results. Currently, only raw mementos from Archive-It are detected and processed. WARC files, of course, are "raw" by their nature, so removing web-archive augmentations like banners is not needed for WARC files.

Measures

OTMT supports the following measures with the -tm (for "timemap measure") option:

• Cosine Similarity of document vectors informed by TF-IDF with scikit-learn (default) (keyword: cosine)
• Word Count (keyword: wordcount)
• Byte Count (keyword: bytecount)
• Simhash on the raw memento content with (keyword: raw_simhash)
• Simhash on the term frequencies of the raw memento content (keyword: tf_simhash)
• Jaccard Distance (keyword: jaccard)
• Sørensen-Dice Distance (keyword: sorensen)
• Cosine similarity of document vectors informed by Latent Semantic Indexing with Gensim (keyword: gensim_lsi)

Each of these measures considers the first memento in a TimeMap to be on-topic and evaluates all other mementos in that TimeMap against that first memento.

Measures and thresholds can be supplied on the command line, separated by commas. For example, to use Jaccard with a threshold of 0.15, separate the measure name and the threshold value, like so:

# detect-off-topic -i archiveit=3936 -o outputfile -tm jaccard=0.15

Multiple measures can also be used, separated by commas. For example, to use jaccard and cosine similarity, type the following:

# detect-off-topic -i archiveit=3936 -o outputfile -tm jaccard=0.15,cosine=0.10

The default thresholds for these measures have been derived from testing using a gold standard dataset of on and off-topic mementos originally generated by AlNoamany. This dataset is now available at: https://github.com/oduwsdl/offtopic-goldstandard-data/. We used this dataset as a standard and selected thresholds that produced the best F₁ score for each measure. I will present the details of how we arrived at these thresholds at iPres 2018. Our study is available as a preprint available on arXiv.

Other options

Optionally, one may also change the working directory (-d) and the logging file (-l). By default, the software uses the directory /tmp/otmt-working for its work and logs to the screen with stdout.

The Future

I am still researching several features that will make it into future releases. I have separated the capabilities into library modules for use with future Python applications, but the code is currently volatile and I expect changes to come in the following months as new features are added and defects are fixed.

The software does not currently offer an algorithm utilizing the Web-based kernel function specified in AlNoamany's paper. This algorithm augments terms from the memento with terms from search engine result pages (SERPs), pioneered by Sahami and Heilman. Due to the sheer number of mementos to be evaluated by the OTMT and Google's policy on blocking requests to its SERPs, I will likely not implement this feature unless it is requested by the community.

I am also interested in the concept of "collection measures". I created the "timemap measures" key in the JSON output to differentiate one set of measure results from another eventual category of collection-wide measures that would test each memento against the topic of an entire collection. Preliminary work using the Jaccard Distance in this area was not fruitful, but I am considering other ideas.

The Off-Topic Memento Toolkit is available at https://github.com/oduwsdl/off-topic-memento-toolkit. Please give it a try and report any issues encountered and features desired. Although developed with an eye toward Archive-It collections, we hope to increase its suitability for all themed collections of archived web pages, such as personal collections created with webrecorder.io.



-- Shawn M. Jones

At iPres 2018, I will be presenting "The Many Shapes of Archive-It", a paper that focuses on some structural features inherent in Archive-It collections. The paper is now available as a preprint on arXiv.

As part of the data gathering for "The Many Shapes of Archive-It", and also as part of the development the Off-Topic Memento Toolkit, I had to write code that extracts metadata and seeds from public Archive-It collections. This capability will be useful to several aspects of our storytelling and summarization work, so I used the knowledge gained from those projects and produced a standalone Python library named Archive-It Utilities (AIU). This library is currently in alpha status, but is already being used with upcoming projects.

The metadata available from an Archive-It collection

Archive-It curators can use the predefined metadata fields of Dublin core. They can also supply their own custom metadata fields.

An screenshot of Archive-It collection 4515 with metadata annotated.

Above is Archive-It collection 4515, named 2013 BART Strike and collected by the San Francisco Public Library. This collection's curators generated quite a bit of metadata. In this screenshot, we can see the following metadata fields for the collection:

• Subject
• Creator
• Publisher
• Source
• Format
• Rights
• Language
• Collector

In addition to collection-wide metadata, we see that the first seed has the following metadata applied:

• Creator
• Publisher
• Language
• Format
• Date

For research purposes, there is quite a lot of data here to be analyzed, especially when comparing collections as we did in "The Many Shapes of Archive-It". I discovered that most collections used the controlled vocabulary from Dublin Core, shown as blue in the bar chart below, more often than freeform vocabulary, shown in green.

Distribution of the top 20 collection-wide metadata fields in public Archive-It collections.

Each collection can have one or more topics. As shown in the screenshot below, the curator can choose from the controlled vocabulary offered by the collection topics field. They can also add their own freeform topics in the subject field. The public-facing interface combines entries from both of these input fields into the public-facing subject field.

Metadata can be added by curators by using the metadata page of one of their Archive-It collections.

The bar chart below shows the distribution of the top 20 topics in public Archive-It collections. I discovered that most curators apply the controlled vocabulary topics to their collections.

Distribution of the top 20 collection-wide subjects (also called topics) of public Archive-It Collections.

This creates a confusing nomenclature. When viewing an Archive-It collection from the outside, everything is displayed as part of the subject field. Because of this, the rest of this post, and Archive-It Utilities, uses the subject field to refer to these topics.

As work for "The Many Shapes of Archive-It" progressed, we focused more on collecting seed lists and then mementos for further analysis. We tried to predict the topics using machine learning, but were unsuccessful and chose a different path for predicting the semantic categories of a collection. Most of the metadata gathered did not make it into the study's results, but will be used in future work. I have included these results here to show the kinds of questions one can answer with Archive-It Utilities.

Installation

Archive-It Utilities requires Python 3.6. Once that requirement has been met, you can install it using:

pip install aiu

It provides several experimental executables. We will only cover fetch_ait_metadata in this post.

Running fetch_ait_metadata

The fetch_ait_metadata command produces a JSON file containing all of the information available about a public Archive-It collection.

To run it on collection 4515 and store the results in file output.json, type the following command:

fetch_ait_metadata -c 4515 -o output.json

The -c option allows one to specify an Archive-It collection and the -o option allows one to indicate where to store the JSON output.

The JSON output looks like the following, truncated for brevity:

{ "id": "4515", "exists": true, "metadata_timestamp": "2018-07-01 16:49:39", "name": "2013 BART Strike", "uri": "https://archive-it.org/collections/4515", "collected_by": "San Francisco Public Library", "collected_by_uri": "https://archive-it.org/organizations/160", "description": "News articles and documents issued by BART regarding the strike of its workers that began on July 1, 2013", "subject": [ "Government", "Government - Cities", "Government - Counties", "2013 BART strike" ], "archived_since": "Apr, 2014", "private": false, "optional": { "creator": [ "Bay Area Rapid Transit" ], "publisher": [ "Bay Area Rapid Transit" ], "source": [ "http://www.bart.gov" ], "format": [ "Web pages and PDFs" ], "rights": [ "Public Domain" ], "language": [ "English" ], "collector": [ "San Francisco Public Library" ] }, "seed_metadata": { "seeds": { "http://www.bart.gov/news/articles/2013/news20130617": { "collection_web_pages": [ { "title": "Sign up for BART labor strike alerts", "creator": [ "San Francisco Bay Area Rapid Transit District" ], "publisher": [ "San Francisco Bay Area Rapid Transit District" ], "language": [ "English" ], "format": [ "Web page" ], "date": [ "August 14, 2013" ] } ], "seed_report": { "group": "", "status": "True", "frequency": "NONE", "type": "normal", "access": "True" } }, ...

From this JSON we can see the name of the collection, which organization created it from the collected_by field, the subjects the curator applied to the collection as a list in the subject field, and when the collection was created in the archived_since field.

Within the optional dictionary field, we see values for freeform metadata added by the curator. In this case we have creator, publisher, source, format, rights, language, and collector.

Also included is the "seed metadata" section containing a list of seeds both scraped from the HTML of the Archive-It collection's web pages and also gathered from the CSV report provided for each Archive-It collection. Above I've listed the seed http://www.bart.gov/news/articles/2013/news20130617 to demonstrate the type of metadata that can be gathered. As noted in "The Many Shapes of Archive-It", seed metadata is optional, but in this example the curator added a title, creator, publisher, language, format, and date to this seed.

Using Archive-It Utilities In Python Code

This information can also be acquired programmatically using the ArchiveItCollection object. The script below demonstrates how one can acquire and the collection name, collecting organization, and the list of seed URIs for Archive-It collection ID 4515.

from aiu import ArchiveItCollection aic = ArchiveItCollection(4515) print( "Collection Name: {}".format( aic.get_collection_name() ) ) print( "Collected By: {}".format( aic.get_collectedby() ) ) print( "Subjects: {}".format( aic.get_subject() ) ) print() print( "List of Seed URIs:" ) for urir in aic.list_seed_uris():     print(urir)

which produces the following output, truncated for brevity:

Collection Name: 2013 BART Strike Collected By: San Francisco Public Library Subjects: ['Government', 'Government - Cities', 'Government - Counties', '2013 BART strike'] List of Seed URIs: http://blog.willamette.edu/mba/ http://blog.willamette.edu/worldnews/ http://blog.willamette.edu/~llc_chinese/ http://blog.willamette.edu/~llc_classics/ http://blog.willamette.edu/~llc_french/ http://blog.willamette.edu/~llc_german/ http://blog.willamette.edu/~llc_japanese/ http://blog.willamette.edu/~llc_russian/ http://blog.willamette.edu/~llc_spanish/ http://blog.willamette.edu/~llc_tellus/ ...

The following methods of the ArchiveItCollection class are useful for analyzing the metadata of a collection:

• get_collection_name - returns the name of the collection
• get_collection_uri - returns the URI of the collection
• get_collectedby - returns the name of the collecting organization
• get_collectedby_uri - returns the URI of the collecting organization
• get_description - returns the content of the "description" field
• get_subject - returns a Python list containing the subjects applied to the collection
• get_archived_since - returns the content of the "archived since" field
• is_private - returns True if the collection is not public, False otherwise
• does_exist - not all collection identifiers are valid, this method returns True if the collection identifier actually represents a real collection, False otherwise
• list_seed_uris - returns a Python list of seed URIs
• get_seed_metadata(uri) - returns a Python dictionary containing metadata for a specific seed at uri
• return_collection_metadata_dict - returns a Python dictionary containing all collection-wide metadata
• return_seed_metadata_dict - a Python dictionary containing all seeds and their metadata
• return_all_metadata_dict - a Python dictionary containing all collection-wide and seed metadata
• save_all_metadata_to_file(filename) - writes all collection-wide and seed metadata out as JSON to a file named filename

The code does perform some measure of lazy loading to be nice to Archive-It. If you only need the general collection-wide metadata, it only acquires the first page of the collection. If you need all seed URIs, it must download all Archive-It pages belonging to the collection.

Summary

Archive-It collections have metadata that can be used to answer many research questions. After working on "The Many Shapes of Archive-It", to be presented at iPres 2018, I used the lessons learned to create Archive-It utilities as a Python library that can be used to acquire this metadata. Please try it out and log any issues at the GitHub repository https://github.com/oduwsdl/archiveit_utilities.

--Shawn M. Jones

(Previous semester highlights posts: Spring 2017, Spring 2016, Spring 2015, Spring/Fall 2013, Fall 2012, Fall 2011)

Here are a few projects that I'd like to highlight from Fall 2017. (All class projects are listed in my InfoVis Gallery.)  All of the projects were implemented using the D3.js library.

World Leader Interactions on Social Media (Twitter) 
Created by Grant Atkins
This project (available at http://www.cs.odu.edu/~gatkins/world-leader-vis/app/) provides an interactive dashboard to visualize ways Twitter list data can be used and represented. This visualization uses the World Leaders list on Twitter, with the addition of a few world leaders not on the list, to derive information and visualize shared information among these users. The goal of this visualization is to show shared term usage among world leaders, see which times tweets are more likely to be sent out, the sentiment of the users, and the decay of data allocated in a static decreasing time interval.


Investigation Into Cryptocurrency Pricing Patterns With Respect to Financial Instability
Created by Jason Orender

This project (available at http://www.cs.odu.edu/~jorender/cs725/CS725-PROJECT_Y9FziL/) provides a focused presentation of the world events co-incident with spikes in peer-to-peer cryptocurrency transactions together with a continuous evolutionary timeline to provide perspective regarding the state of development and usage at a national, regional and worldwide level. Bitcoin was the sole cryptocurrency used in this analysis due to the large amount of country specific peer-to-peer data available.



Holiday Flight Patterns

This project (available at http://www.cs.odu.edu/~agosav/cs725/HolidaysFlightPattern/index.html) is an interactive tool for visualizing holiday flight patterns using a dot chart to visualize the last year's data, a US map with bubbles to display the average arrival delay at particular airports, and a line chart which shows the monthly distribution from 2006-2015. The datasets selected for this visualization are the percentage of on-time arrival, delayed and cancellations for different airlines operating in the US, over different years. The intention was to find flight delay patterns between holiday and non-holiday months.


This project (available at http://www.cs.odu.edu/~jashley/cs725/project/) highlights some of the characteristics that make up the current Federal civil service workforce. The visualization also provides a visual snapshot of how widely dispersed the workforce is.

Video available at http://www.cs.odu.edu/~jashley/cs725/demo/
-Michele

On July 13, I saw this on Twitter:

On Monday, doctors, nurses and researchers will lose access to a trove of medical data, as the Trump Administration shuts down the National Guidelines Clearinghouse. See our piece today, by our Senior Investigator @j0ncampbell, in @TheDailyBeast: https://t.co/Qz1tkRIQIM

— Sunlight Web Integrity Project (@SunWebIntegrity) July 12, 2018

There are two US government websites in danger, the National Guideline Clearinghouse (https://www.guideline.gov) and the National Quality Measures Clearinghouse (https://qualitymeasures.ahrq.gov). Both store medical guidelines. Both will "not be available after July 16, 2018". According to the linked Daily Beast article above:

Medical guidelines are best thought of as cheatsheets for the medical field, compiling the latest research in an easy-to-use format. When doctors want to know when they should start insulin treatments, or how best to manage an HIV patient in unstable housing — even something as mundane as when to start an older patient on a vitamin D supplement — they look for the relevant guidelines. The documents are published by a myriad of professional and other organizations, and NGC has long been considered among the most comprehensive and reliable repositories in the world.

The Sunlight Foundation Web Integrity Project wrote a report about the archivabilty of this service. They note that "interactive features do not function, making archived content much more difficult to access and, in many cases completely unavailable." Seeing as web archives typically crawl websites from the client side and have no access to the server components, I expect that the search functionality of the web sites should not work once archived.

The robots.txt for www.guideline.gov disallows everyone:

User-agent: * Disallow: /
The robots.txt for qualitymeasures.ahrq.gov disallows everyone:

User-agent: * Disallow: /

In December of 2016, the Internet Archive stopped honoring robots.txt for .gov and .mil websites, hoping to "keep this valuable information available to users in the future". Seeing at these two sites will be shut down on July 16, 2018, how well are they archived?

Experiment Setup

The method I used to evaluate how much of each site was archived consisted of the following general steps:

• Acquire a sample of original resource URIs from www.guideline.gov and qualitymeasures.ahrq.gov
• Use a Memento Aggregator to determine if each original resource has at least one memento

As we can see in the robots.txt above, there is no machine readable site map for either web site. This means that I would need to crawl each site to find all of the URIs. Remembering lessons from when I evaluated URI patterns for Signposting the Scholarly Web, and when I manually crawled a number of scholarly web sites looking for URI patterns to help other crawling efforts, I started off thinking like someone who was planning to archive each site. I knew that I did not have time to manually crawl the entirety of each site so I tried to evaluate which documents appeared to be the main products of these sites. I have classified the documents I evaluated into five categories: main products - summaries, expert commentaries, guideline syntheses, summaries in other formats, and other pages. I will describe these categories in more detail in the following sections.

I created a GitHub repository to save my work. Due to the time crunch, I did not organize it nicely and it will be updated in the coming days with more content used in this article, so check back to it often if interested.

Update on 2018/07/16 at 20:37 GMT:The GitHub repository is now as stable as it is going to be. As it was written over the course of 3 days, the code is very, very rough. I have no intentions of improving it, but the data and code is provided for anyone who is interested. Feel free to contact me on Twitter with any questions.

After acquiring sample original resource URIs to test I installed MemGator, a Memento Aggregator developed by the WS-DL Research Group. I wrote a Python script which requested an aggregated TimeMap from MemGator for each original resource URI and recorded the number of mementos per URI.

So, what categories of documents did I retrieve before feeding them into MemGator?

Main Products - Summaries

I reviewed the menus across the top of each site's home page. I discovered that the main product of www.guideline.gov appeared to be the guideline summaries and the main product of qualitymeasures.ahrq.gov appeared to be measure summaries. I focused on these documents because, if captured as mementos, an enterprising archivist could build their own search engine around them.

As shown in the screenshot below, these summaries were accessible via paginated search result pages. Fortunately, there is an "All Summaries" option which will list all summaries as a series of search results.


The qualitymeasures.ahrq.gov site also has its own "All Summaries" page, shown in the screenshot below, so these URIs can be scraped using a script aware of the paging as well.



As Corren wrote last year, pagination can result in a missed captures. Knowing this, I wondered if the pagination would have an impact on if the guideline summaries were archived.

I wrote some simple (and very rough) code in Python using the requests library and BeautifulSoup to scrape all URIs from each search result page. The same script was used to scrape both sites. For both sites I selected the guideline summary URIs, identified because they contained the string "/summaries/summary/", and removed duplicates. This gave me 1415 original URIs for www.guideline.gov and 2533 original URIs for qualitymeasures.ahrq.gov.

Expert Commentaries

Both sites also contained expert commentaries about these summaries. I decided that this also looked important, even though these commentaries did not appear to be indexed by the search engine.

A screenshot of the Expert Commentaries page on www.guideline.gov

A screenshot of the Expert Commentaries page on qualitymeasures.ahrq.gov

I wrote a script to scrape the expert commentary URIs from these pages. With this I ended up with 45 URIs for www.guideline.gov and 52 URIs for qualitymeasures.ahrq.gov.

Guideline Syntheses

The www.guideline.gov site has a series of documents labeled guideline synthesis documenting "areas of agreement and difference, the major recommendations, the corresponding strength of evidence and recommendation rating schemes, and a comparison of guideline methodologies". These documents also seemed to be important, so I chose to include them as well.

The Guideline Synthesis page at www.guideline.gov is another set of documents provided by the web site.

I wrote a script to scrape this page for all guideline synthesis URIs. This led me to 18 URIs for www.guideline.gov. The qualitymeasures.ahrq.gov site did not contain this type of document.

Summaries in other formats

In addition to the HTML formatted guideline summaries, there were guideline summaries available in PDF, XML, and DOC format on www.guideline.gov. I wrote another script to iterate through all of the summary pages captured in the previous section and save off the PDF, XML, and DOC URIs. The qualitymeasures.ahrq.gov website only has HTML formatted measure summaries, so this document category does not apply to that site.

This screenshot demonstrates the multiple formats available for a guideline summary on www.guideline.gov.

My script to scrape these pages gave me 4185 URIs for www.guideline.gov.

Other Pages

Finally, I was curious about what may have been missed elsewhere. I decided to try to gather URIs as a crawler would who is given the seed of the top level page. With this exercise, I was hoping to gather a number of top level pages to see how their archive status differed from the guideline summaries, expert commentaries, guideline syntheses, and the measure summaries.

I wrote two simple spiders (crawlers) using the Python crawling framework scrapy. I pointed each spider at the homepage of each website, instructed it not to crawl outside of the domain of each site, and told it to print out any URI listed on a page it discovered while crawling. Unfortunately, I ran it on a machine with insufficient memory. The operating system killed scrapy in both cases because it was consuming too much memory. This means that the crawl for www.guideline.gov ran for 4 hours while the crawl for qualitymeasures.ahrq.gov ran for 7 hours. This inconsistency in crawl times was disappointing, but I kept the URIs from these crawls because they provide an interesting contrast in the results section.

Once I had a list of URIs linked from pages encountered during the crawl, I then removed all URIs that were not in the domains of www.guideline.gov or qualitymeasures.ahrq.gov, respectively.

Hundreds of thousands of URIs returned were related to search facets. The crawl of www.guideline.gov returned 894,881 such URIs while the crawl of qualitymeasures.ahrq.gov returned 1,474,516. Because these search facet URIs were related to the summaries from the prior sections, I removed these search URIs in the interest of time and only focused on the other pages crawled because these other pages contained actual content. I removed any URIs containing fragments (i.e. hashes like #introduction). I also filtered the URIs for summaries, guideline syntheses, and expert commentaries so that there would be no overlap in results.

I then fed the URIs through MemGator to see if the pages were captured.

Results and Discussion

The table below shows the results of testing if a page was archived for www.guideline.gov. Of those URIs recorded for this experiment, 98.8% of them were indeed archived, which is good news.

www.guideline.gov Page Category	Archived	Not Archived	Total
Guidelines Summaries (HTML)	1401	14	1415
Expert Commentaries	45	0	45
Guideline Syntheses	18	0	18
Guideline Summaries (Other Formats)	4185	57	4242
Other Pages	150	2	152
Total	5799 (98.8%)	73 (1.2%)	5872

Most importantly, of the 1415 guideline summaries from www.guideline.gov, 1401/1415 (99.0%) are archived. Only 14/1415 (1.0%) are not archived. Also, all 45 expert commentaries and all 18 guideline syntheses are archived. This means that almost all of the important site content is preserved and an enterprising archivist can build a search engine around them in the future.

The table below shows the results of testing if a page was archived for qualitymeasures.ahrq.gov. Of the URIs recorded for this experiment, 97.5% of them were archived.

qualitymeasures.ahrq.gov Page Category	Archived	Not Archived	Total
Measures Summaries	2509	24	2533
Expert Commentaries	52	0	52
Other Pages	90	44	134
Total	2651 (97.5%)	68 (2.5%)	2719

Of the 2533 measure summaries from qualitymeasures.ahrq.gov, 2509/2533 (99%) are archived. Only 24/2533 (0.9%) were not archived. Also, all 52 expert commentaries are archived. Again, this means that the majority of the important documents exist in a web archive and can be indexed by a potential search engine in the future. The picture is not so good for the other pages category, where only 90/134 (67.2%) of the pages exist in a web archive.

The high overall numbers are remarkable and likely a result of the Internet Archive's efforts to remove the robots.txt restrictions at the end of 2016. The next sections answer additional questions.

What is the distribution of mementos per category per site?

Below several histograms show the distribution of memento counts across the the different categories of pages for www.guideline.gov. Note that this only applies to those pages with mementos.

Histogram of the number of mementos per URI for guideline summaries for www.guideline.gov. Minimum: 1, Maximum: 24, Mode: 8. Note: only pages with mementos were evaluated.

Histogram of the number of mementos per URI for expert commentaries for www.guideline.gov. Minimum: 9, Maximum: 14, Mode: 11. Note: only pages with mementos were evaluated.

Histogram of the number of mementos per URI for guideline syntheses for www.guideline.gov. Minimum: 9,  Maximum: 18, Mode: 14. Note: only pages with mementos were evaluated.

Histogram of the number of mementos per URI for non-HTML guideline summaries for www.guideline.gov. Minimum: 1, Maximum: 13, Mode: 9. Note: only pages with mementos were evaluated.

Histogram of the number of mementos per URI for other pages for www.guideline.gov. Minimum: 1, Maximum: 2072, Mode: 1. Note: only pages with mementos were evaluated.

We see that the more specific content in the guideline summaries, expert commentary, guideline syntheses, and the guideline summaries in multiple formats tend to have a mode of 8, 9, 11, or 14 mementos. This means that many of the more important pages have multiple mementos. The other content, consisting mostly of top level pages, has a mode of 1, meaning that a lot of these top level pages were only archived once. There is at least one page in the other category, though, that was archived 2072 times.

Below several histograms show the distribution of memento counts across the the different categories of pages for qualitymeasures.ahrq.gov.

Histogram of the number of mementos per URI for measure summaries for qualitymeasures.ahrq.gov. Minimum: 1, Maximum: 15, Mode: 4.

Histogram of the number of mementos per URI for expert commentaries for qualitymeasures.ahrq.gov. Minimum: 6, Maximum: 7, Mode: 7.

Histogram of the number of mementos per URI for other pages for qualitymeasures.ahrq.gov. Minimum: 2, Maximum 131, Mode: 2.

The numbers are much lower for qualitymeasures.ahrq.gov, but they exhibit the same pattern.

How does the crawling pattern for mementos change over time per category per site?

So, how does the crawling of www.guideline.gov change over time? The bar charts below show the number of mementos added to archives per month based on their memento-datetime.

Memento count per month for guideline summaries for www.guideline.gov. We see a big push in more recent months.

Memento count per month for expert commentaries for www.guideline.gov. There is much the same pattern as for the prior category.

The number of mementos crawled per month for the guideline syntheses  documents of www.guideline.gov. There has been a lot of activity the past few months.

Memento count per month for the non-HTML versions of guideline summaries for www.guideline.gov. Again, we see a big push in more recent months.

Memento count per month for other pages at www.guideline.gov. Here we see years of crawling with big spikes after the US election. This may be related to the Internet Archive's new robots.txt policy.

It is interesting to note that mementos exist for some of these pages prior to December of 2016, meaning that people were archiving them with functionality such as "Save Page Now". Archiving really picked up in all cases in September of 2016, then again in October of 2017, and then again starting in April of 2018. These spikes appear to be a coordinated effort to archive parts of the site.

In the last graph, we see years of crawling the top level pages. This is interesting considering the contents of the robots.txt file. Did it change over time? Was it more permissive at some point? Fortunately, we have web archives we can use to check.

Here is a screenshot of the Internet Archive's capture calendar for www.guideline.gov/robots.txt from 2005. Orange indicates that the robots.txt file did not exist. Blue indicates that it did.

Based on the above screenshot, it appears that a robots.txt did not exist for the site www.guideline.gov until 2005. It was first observed at this site on August 23, 2005 at 22:54:19 GMT. Its contents were as follows:
User-agent: * Disallow:

According to the robots.txt specification website, this indicates "To allow all robots complete access". This means that at one time, the site was far more permissive about crawling than it is now. I randomly chose a memento for the robots.txt each year after 2005 and found that it did not change. In August of 2008, the robots.txt disappeared again. In 2009, the successful robots.txt captures are actually of a soft-404 page indicating that it does not exist. Before September 11, 2010 at 18:14:50 GMT, the robots.txt became more complex, as shown below:

User-agent: * Disallow: /bin/ Disallow: /config/ Disallow: /css/ Disallow: /radControls/ Disallow: /images/ Crawl-delay: 30 # number of seconds to wait between successive requests to the same server Request-rate: 1/15 # maximum rate is one page every 15 seconds Visit-time: 0500-1000 # only visit between 05:00 and 10:00 UTC (GMT) (00:00-05:00 EST/01:00-6:00 EDT)

As we see, it still isn't disallowing all content like I mentioned at the beginning of the article. This configuration persisted until August 26, 2016 when the robots.txt was still present, but a completely blank file. The robots.txt was changed to its current state on April 27, 2017 before 20:09:28 GMT. The US Senate approved the nomination of Tom Price to the office of Secretary of Health and Human Services on February 1, 2017. This means that the site's robots.txt allowed crawling until after Tom Price took office. This is probably why so many top level pages had been captured by web archives since the site's creation.

What about the qualitymeasures.ahrq.gov site? The bar charts below show the number of mementos per month for each of its categories.

Memento count per month for measure summaries at qualitymeasures.ahrq.gov. There is some activity in 2016, but a lot of very recent crawling of the content.

Memento count per month for measure summaries at qualitymeasures.ahrq.gov. Like above, there is some activity in 2016, but a big push in June of 2017, and a lot of very recent crawling of the content.

Memento count per month for measure summaries at qualitymeasures.ahrq.gov. We see the same large push in recent history, with a lot of crawling.

The crawling of qualitymeasures.ahrq.gov follows much the same pattern, though not with the exact same spikes prior to this last month. From these graphs we see that there has been a concerted effort to archive both of these sites since June. This site created its first robots.txt on August 24, 2005 before 00:04:23 GMT. And the robots.txt was completely permissive, as with www.guideline.gov.

The emergence of a robots.txt for qualitymeasures.ahrq.gov on August 24, 2005, as shown on the Internet Archive's calendar page for the URI qualitymeasures.ahrq.gov/robots.txt.

The robots.txt went through much the same history for this site as for www.guideline.gov, implying a similar policy or even the same webmaster for both sites. It finally changed to its current disallow state on April 27, 2017 before 22:10:11 GMT. Again, this is after Tom Price took office. This again explains why so many of the top level pages of the site were archived throughout the history of qualitymeasures.ahrq.gov.

In which archives are these pages preserved?

I chose to use an aggregator because I wanted to search in multiple web archives for these pages. How well do these mementos spread throughout the archives? The charts below show the number of mementos per archive for each category of pages at www.guideline.gov. Only archives containing mementos for a given category are displayed in each chart.

This chart of the guideline summaries for www.guideline.gov shows 6,848 mementos are present in the Internet Archive, with 4,887 mementos preserved by Archive-It and 10 mementos preserved by Archive.today (archive.is).

This chart of the non-HTML versions of the guideline summaries for www.guideline.gov shows 14,846 mementos are  preserved in the Internet Archive, but more, 19,044 are preserved in Archive-It.

This chart of the expert commentaries for www.guideline.gov shows that 324 mementos are held by the Internet Archive while 177 are held by Archive-It.

This chart of the guideline syntheses for www.guideline.gov shows that 189 mementos are held by the Internet Archive while 62 are held by Archive-It.

The chart of the other pages for www.guideline.gov shows that the top-level pages are preserved at more archives than the previous categories. There are 3,397 mementos at the Internet Archive, 819 mementos at Archive-It, 128 mementos at the Library of Congress, 19 at Archive.today, 11 at the Icelandic Web Archive, 7 at the Portuguese Archive, and 1 at Perma.cc.

While the Internet Archive and Archive-It have most of the mementos, some mementos of the top-level pages of the site are held in other archives. As this is a US government web site, I was surprised that the Library of Congress was not featured more. Archive-It also has more non-HTML guideline summaries than the Internet Archive, indicating a particular effort by some organization to preserve these documents in other formats. Unfortunately, the Archive-It mementos I discovered with MemGator belonged to the collection /all/ meaning that I have no indication as to which Archive-It collection or organization was preserving the pages.

Update on 2018/07/16 at 18:10 GMT: To find the specific Archive-It collection and collecting organization, Michele Weigle has suggested that one might be able to search the Archive-It collections for these URIs using Archive-It's explore all archives search interface. One would need to use the "Search Page Text" tab. I did try the string www.guideline.gov and discovered 5,906 search results, so this hostname is in the content of some of these pages. I tried using a URI reported to have an Archive-It memento, but did not receive any search results. If you are successful, please say something in the comments.

The bar charts below show the distribution of mementos across web archives for the qualitymeasures.ahrq.gov web site.

This chart of the measure summaries for qualitymeasures.ahrq.gov shows 9,494 mementos at the Internet Archive and only 216 at Archive-It.

This chart of the measure summaries for qualitymeasures.ahrq.gov shows all 360 mementos of expert commentaries are held by the Internet Archive.

This chart for the other pages at qualitymeasures.ahrq.gov shows 1,147 mementos at the Internet Archive, 145 mementos at Archive-It, 40 mementos at the Library of Congress, 12 at the Portuguese Web Archive, 1 at Archive.today, and 1 at Perma.cc.

The results for qualitymeasures.ahrq.gov show that most of the mementos for that site are archived at the Internet Archive, with a few in other archives. This is in contrast to the results for www.guideline.gov, where the numbers between the Internet Archive and Archive-It were close in many cases.

Attempts at Archiving the Missing Pages

On July 14, 2018, I attempted to use our own ArchiveNow to preserve the ~1% of summary URIs from each site that had not been archived. Unfortunately, the live resources started responding very slowly. The sample of summary URIs that had not been archived produced 500 status codes, as can be seen in the output from the curl commands below, each which took close to a minute to execute:

# curl -I "https://www.guideline.gov/summaries/summary/51259/acr-appropriateness-criteria--chronic-ankle-pain" HTTP/1.1 500 Internal Server Error Cache-Control: private Content-Length: 276 Content-Type: text/html; charset=utf-8 Server: Microsoft-IIS/10.0 Set-Cookie: ASP.NET_SessionId=oppgaow0ikkgopcfbk0mnusm; path=/; HttpOnly Set-Cookie: ASP.NET_SessionId=oppgaow0ikkgopcfbk0mnusm; path=/; HttpOnly Set-Cookie: NGC=RecentlyViewedContent=51259,51259; expires=Mon, 13-Aug-2018 16:45:04 GMT; path=/; HttpOnly X-AspNetMvc-Version: 4.0 X-AspNet-Version: 4.0.30319 Request-Context: appId=cid-v1:deb297e9-23b8-4f6e-b376-44111ebc4951 X-Powered-By: ASP.NET X-Frame-Options: SAMEORIGIN Content-Security-Policy: frame-ancestors 'none'; Set-Cookie: ARRAffinity=da7985c62a729352f6a2745ce1af0df3e542bf945932bc766fb35d7c851d7626;Path=/;HttpOnly;Domain=www.guideline.gov Date: Sat, 14 Jul 2018 16:46:20 GMT Strict-Transport-Security: max-age=31536000; includeSubDomains;

# curl -I "https://qualitymeasures.ahrq.gov/summaries/summary/51246/hospice-experience-percentage-of-caregivers-who-reported-that-their-family-member-was-treated-with-respect" HTTP/1.1 500 Internal Server Error Cache-Control: private Content-Length: 276 Content-Type: text/html; charset=utf-8 Server: Microsoft-IIS/10.0 Set-Cookie: ASP.NET_SessionId=erbtndkp131sxor5h2rhocv5; path=/; HttpOnly Set-Cookie: ASP.NET_SessionId=erbtndkp131sxor5h2rhocv5; path=/; HttpOnly Set-Cookie: NQMC=RecentlyViewedContent=51246,51246; expires=Mon, 13-Aug-2018 16:55:45 GMT; path=/; HttpOnly X-AspNetMvc-Version: 4.0 X-AspNet-Version: 4.0.30319 Request-Context: appId=cid-v1:deb297e9-23b8-4f6e-b376-44111ebc4951 X-Powered-By: ASP.NET X-Frame-Options: SAMEORIGIN Content-Security-Policy: frame-ancestors 'none'; Set-Cookie: ARRAffinity=da7985c62a729352f6a2745ce1af0df3e542bf945932bc766fb35d7c851d7626;Path=/;HttpOnly;Domain=qualitymeasures.ahrq.gov Date: Sat, 14 Jul 2018 16:56:43 GMT Strict-Transport-Security: max-age=31536000; includeSubDomains;

I ran curl on all live URIs listed as not captured and they return a HTTP 500 status as of July 14, 2018 at approximately 16:50:00 GMT. Because I had scraped these URIs from the "All Summaries" page, it is possible that they returned 500 statuses at the time of crawl and this is why web archives do not currently have them. This means, that, even on the live web, they were not available. The live versions of the other summary pages with mementos returned a 200 status (after about a minute delay).

It is also possible that the service at these web sites is degrading in their last hours. As of approximately 07:00 GMT on July 15, 2018, the qualitymeasures.ahrq.gov site was no longer available, displaying error messages for pages, as shown in the screenshot below.

As of 07:00 GMT on July 15, 2018, the qualitymeasures.ahrq.gov website started displaying error messages instead of content.

Update on 2018/07/16 at 19:00 GMT: The website qualitymeasures.ahrq.gov is available again, but the measure summaries that were missing from the archives still return HTTP 500 status codes. The missing guideline summaries for www.guideline.gov also still return HTTP 500 status codes.

This was quite disheartening, because my plan was to archive the pages I had detected as missing after I did my initial study. I thought I had until July 16 to save the web pages!

Conclusion

Almost all web archiving is done externally, with no knowledge of the software running on the server side. This reduces mementos to a series of observations of pages rather than a complete reproduction of all of the functionality that existed at a web site. In the case of two US government websites that will be shut down on July 16, 2018, www.guideline.gov and qualitymeasures.ahrq.gov have server-side functionality, but their most valuable assets are a series of summary documents that can be captured without having to reproduce the functionality of the server side. In this article, I've tried to determine how much of these web sites have been captured prior to their termination.

When focusing on the main products of each site, the guideline summaries and the measure summaries, we see that these products are actually pretty well archived, at 99% of guideline summaries for www.guideline.gov and 99% of measure summaries for qualitymeasures.ahrq.gov. We also observed that 100% of all expert commentaries were archived in both cases. Other aspects of the site, such as trying to reproduce all facets of the search engine were not tested. I did, however, attempt to crawl the sites to gain a list of pages outside of these categories and discovered that, at least among the pages captured during a limited crawl, other pages at www.guideline.gov are archived at a percentage of 99%, higher than those for qualitymeasures.ahrq.gov, which only stand at 67.2%.

Many of these main products have more than one memento and as many as 25 in some cases. There are more mementos for www.guideline.gov than for qualitymeasures.ahrq.gov, but the mode for the number of mementos of the main products range between 4 and 14 mementos. This means that the main products have good coverage. The top-level content at these sites, however, has a mode of 1 or 2 mementos, indicating poor coverage of the changes over time for some top-level pages.

Over the life of these sites, most of the mementos stored in web archives are for the top-level pages, because crawling was permitted by their robots.txt until April 27, 2017, a few months after Tom Price became the Secretary of Health and Human Services. Fortunately, there has been a large push to archive the main products of the site since September of 2016, resulting in many mementos created within the last month.

Most of the mementos for these sites are stored in the Internet Archive. Archive-It has more mementos of the non-HTML versions of guideline summaries for www.guideline.gov, but its memento count is eclipsed by the Internet Archive in all other cases. After the Internet Archive and Archive-It, there is a long tail of archives for top-level pages, but the number of mementos for each of these archives is less than 100. With the exception of 10 guideline summaries for www.guideline.gov stored in Archive.today, none of the main products of these sites are stored outside of the Internet Archive or Archive-It.

My attempts to archive the pages after running this experiment failed, in large part due to the degradation in service at these web sites. Even though I tried preserving the pages prior to the cutoff date of July 16, 2018, they were no longer reliably available.

Because one needs to know the original resource URI in order to find mementos in a web archive, I have published the URIs I discovered to Figshare. I do this in hopes that someone might build a resource for providing easy access to the content of these sites, especially for medical personnel. If you want to access them, use these links.

• the URIs discovered for www.guideline.gov
• the URIs discovered for qualitymeasures.ahrq.gov

Feel free to contact me if you run into problems with these files.

This case demonstrates the importance of organizations like the Sunlight foundation for identifying at risk resources. Also important are the web archives for allowing us to preserve these resources. This case also demonstrates how we can come together and ensure that these resources are preserved. We do need to be concerned that so much of this content is preserved in one place, rather than spread across multiple archives. If a page is of value to you, you have an obligation to archive it and archive it in multiple archives. What web pages have you archived today, so that you, and others, can access their content long after the live site has gone away?

--Shawn M. Jones

Google.com mementos from May 8th 1999 on the Internet Archive

In terms of the ability to be archived in public web archives, web pages fall into one of two categories: publicly archivable, or not publicly archivable.

1. Publicly Archivable Web Pages:

These pages are archivable by public archives. The pages can be accessed without login/authentication. In other words, these pages do not reside behind a paywall. Grant Atkins examined paywalls in the Internet Archive for news sites and found that web pages behind paywalls may actually be redirecting to a login page at crawl time. A good example of a publicly archivable page is Dr. Steven Zeil's page since no authentication is required to view the page. Furthermore, it does not use client-side scripts (i.e., Ajax) to load additional content, so what you see in the web browser and what you can replay from public web archives are exactly the same.

Screen shot from Dr. Steven Zeil's page captured on 2018-07-02

Memento for Dr. Zeil's page on the Internet Archive captured on 2017-12-02

Some web pages provide "personalized" content depending on the GeoIP of the requester. In these cases, what you see in the browser and what you can replay from public web archives are nearly the same, except for some minor personalization/GeoIP related changes. For example, a user requesting https://www.islamicfinder.org from Suffolk, Virginia will see the prayer times for the closest major city (Norfolk, Virginia). On the other hand, when the Internet Archive crawls the page, it sees the prayer times for San Bruno, California. This is likely because the crawling/archiving is happening from San Francisco, California. The two pages, otherwise, are exactly the same!

Leaping Tiger statue next to the College of Arts at Towson University

Day 1 (July 9, 2018)