updated documentation after a readthrough

diff --git a/.gitignore b/.gitignore
index 77320b339fe11a618a93b46e70be95d88a52a9e7..5bf2ab9147c8f221d463fa743a7ec474b0cccf7d 100644 (file)
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,3 @@
 output/*
 output/*

+docs/*html
+

diff --git a/README.rst b/README.rst
index 081f765b952335846b6c0b3dc9ddf76dc8e79d1c..f7f4dad2e4eee211d5a9d2c9ce7afc1e6e75c598 100644 (file)
--- a/README.rst
+++ b/README.rst
@@ -1,20 +1,23 @@
 MediaWiki Redirect Tools
 =======================================================================
 
 MediaWiki Redirect Tools
 =======================================================================
 

-  | **Author:** `Benjamin Mako Hill`__ <mako@atdot.cc>
+  | **Author:** `Benjamin Mako Hill`__ <mako@atdot.cc> and `Aaron Shaw`__ <aaron.d.shaw@gmail.com>

   | **Homepage:** http://networkcollectiv.es/wiki-redirects/
   | **Homepage:** http://networkcollectiv.es/wiki-redirects/

-  | **License:** `GNU GPLv3 or any later version`__ (see COPYING)
+  | **Code:** http://projects.mako.cc/source/?p=redirect-tools
+  | **License:** `GNU GPLv3 or any later version`__

   | **Description:** Tools to to generate a redirect spells dataset from "raw" MediaWiki XML dumps like those published by the Wikimedia foundation.
 
 __ http://mako.cc/
   | **Description:** Tools to to generate a redirect spells dataset from "raw" MediaWiki XML dumps like those published by the Wikimedia foundation.
 
 __ http://mako.cc/

+__ http://aaronshaw.org/

 __ http://www.gnu.org/copyleft/gpl.html
 
 __ http://www.gnu.org/copyleft/gpl.html
 

-If you use this software for research, please **cite the following
-paper** in any resulting publication:
+If you use this software for research, please cite the following
+paper:

 
 

-  *Hill, Benjamin Mako and Aaron Shaw. "Consider the Redirect:  A Missing
-  Dimension of Wikipedia Research." In Proceedings of the 10th International
-  Symposium on Open Collaboration (OpenSym 2014). ACM Press, 2014.*
+  *Hill, Benjamin Mako and Aaron Shaw. "Consider the Redirect: A
+  Missing Dimension of Wikipedia Research." In Proceedings of the 10th
+  International Symposium on Open Collaboration (OpenSym 2014). ACM
+  Press, 2014.*

 
 **Overview:**
 
 
 **Overview:**
 


@@ -34,10 +37,10 @@ __ https://simple.wikipedia.org/
 
 Before you start, you may also want to change the default directories for writing intermediate output files.  The default directories for writing and reading files are at the top of the file `redirect_tools.R` and can be changed by editing that file. By default, all files will be written to the subdirectory "./output" in the local directory. If you want to use the default directories, you will still need to create them with a command like this::
 
 
 Before you start, you may also want to change the default directories for writing intermediate output files.  The default directories for writing and reading files are at the top of the file `redirect_tools.R` and can be changed by editing that file. By default, all files will be written to the subdirectory "./output" in the local directory. If you want to use the default directories, you will still need to create them with a command like this::
 

-  mkdir output; output/redir; output/spells
+  mkdir output output/redir output/spells

 
 

-Step 2: Find Redirects in Revisions
-=======================================================================
+Step 1: Find Redirects in Revisions
+-----------------------------------------

 
 Dependencies:
 
 
 Dependencies:
 


@@ -52,9 +55,7 @@ Output:
 
 - bzip2 compressed TSV files (one line per revision)
 
 
 - bzip2 compressed TSV files (one line per revision)
 

-You will run the `01-extract_redirects.py` script to build a dataset of revisions or edits that marks every revisions as either containinig a revision, or not.
-
-Then, the script `01-extract_redirects.py` will take a MediaWiki dump file on STDIN and output a TSV file on STDOUT of the following form.
+You will run the `01-extract_redirects.py` script to build a dataset of revisions or edits that marks every revisions as either containing a redirect, or not. `01-extract_redirects.py` takes a MediaWiki dump file on STDIN and output a TSV file on STDOUT of the following form:

 
 +---------+-------------+--------------------------------+------------+---------+----------+--------------------+
 | page.id | revision.id | page.title                     | timestamp  | deleted | redirect | target             |
 
 +---------+-------------+--------------------------------+------------+---------+----------+--------------------+
 | page.id | revision.id | page.title                     | timestamp  | deleted | redirect | target             |


@@ -66,7 +67,7 @@ Then, the script `01-extract_redirects.py` will take a MediaWiki dump file on ST
 
 In this (example) case, the first revision of the article "Mikhail Alekseevich Lavrentiev" was not a redirect but the second is a redirect to "Mikhail Lavrentyev."
 
 
 In this (example) case, the first revision of the article "Mikhail Alekseevich Lavrentiev" was not a redirect but the second is a redirect to "Mikhail Lavrentyev."
 

-If you are using the Simple English dump (which is a single file) you would run the following command to send the output to the default destination::
+If you are using the Simple English dump (which is a single file) you would run the following command to uncompress the dump, parse it using our script, compress the output again, and save the output to the default destination::

 
   7za x -so simplewiki-20140410-pages-meta-history.xml.7z | 
   python2.7 01-extract_redirects.py | bzip2 -c - > output/redir/simple_redirs.tsz.bz2
 
   7za x -so simplewiki-20140410-pages-meta-history.xml.7z | 
   python2.7 01-extract_redirects.py | bzip2 -c - > output/redir/simple_redirs.tsz.bz2


@@ -75,13 +76,13 @@ Because our dumpfile is 7z compressed, I used 7za to uncompress it. If I had use
 
 
 Step 2: Generate spells
 
 
 Step 2: Generate spells

-=======================================================================
+-----------------------------------------

 
 Dependencies:
 
 - GNU R
 - data.table (http://cran.r-project.org/web/packages/data.table/)
 
 Dependencies:
 
 - GNU R
 - data.table (http://cran.r-project.org/web/packages/data.table/)

-- foriegn (http://cran.r-project.org/web/packages/foreign/)
+- foreign (http://cran.r-project.org/web/packages/foreign/)

 
 Input: 
 
 
 Input: 
 


@@ -105,23 +106,31 @@ You can run the command with::
 
   R --no-save < 02-generate_spells.R
 
 
   R --no-save < 02-generate_spells.R
 

-By default, output will be saved into `output/spells`.
+By default, output will be saved into `output/spells`. The script will
+save three versions of the output:

 
 

-The 
+1. `redirect_spells.RData` — An RData file suitable for use in GNU R
+2. `redirect_spells.tsv` — A tab seperated values file suitable for use in a variety of different programs.
+3. `redirect_spells.dta` — A DTA file suitable for use in Stata (many versions will crop very long artiicle titles due to limitations in the DTA format).

 
 

-Running Code in Parallel
-=======================================================================

 
 

-Because the full history dumps from the WMF foundation are split into many
-files, it is can be appropriate to parse these dumps in parallel. Although the
-specific ways you choose to do this will vary by the queuing system you use,
-we've included examples of the scripts we used with Condor on the Harvard/MIT
-Data Center (HMDC) in the "examples/" directory. They will not work without
-modification for your computing environment because they have our environment
-hardcoded in but they will give you an idea of where you might want to start.
-
-Additionally, there is a third step `03-assemble_redirect_spells.R` that
-contains R code that will read in all of the separate RData files, assmebles
-the many smaller dataframes into a single data.frame, and then saves that
-unified data.frame into a single RData file.
+Running Code in Parallel
+-----------------------------------------
+
+Because the full history dumps from the WMF foundation are split into
+many files, it is usually appropriate to parse these dumps in
+parallel. Although the specific ways you choose to do this will vary
+by the queuing or scheduling system you use, we've included examples
+of the scripts we used with Condor on the Harvard/MIT Data Center
+(HMDC) in the `examples/` directory of the source code. They will not
+work without modification for your computing environment because they
+have configuration options and paths for our environment
+hardcoded. That said, they may give you an idea of where you might
+want to start.
+
+In this parallel code there is a third file
+`03-assemble_redirect_spells.R` that contains R code that will read in
+all of the separate RData files created in paralell processing,
+assemble the many smaller dataframes into a single dataframe, and then
+saves that unified data.frame into a single RData file.

 
 

diff --git a/docs/README.rst b/docs/README.rst
new file mode 120000 (symlink)
index 0000000..89a0106
--- /dev/null
+++ b/docs/README.rst
@@ -0,0 +1 @@
+../README.rst
\ No newline at end of file

diff --git a/docs/homepage.rst b/docs/homepage.rst
index 206afa84299954a426ef1701d74e29f1eb9f1ca7..a0f8ad66ec47620ffb12c4f09fbf071bd654f376 100644 (file)
--- a/docs/homepage.rst
+++ b/docs/homepage.rst
@@ -14,27 +14,28 @@ __ https://en.wikipedia.org/wiki/Main_Page
 __ https://en.wikipedia.org/wiki/Seattle,_Washington
 __ https://en.wikipedia.org/wiki/Seattle
 
 __ https://en.wikipedia.org/wiki/Seattle,_Washington
 __ https://en.wikipedia.org/wiki/Seattle
 

-In wikis, **redirects** are special pages in that silently take readers
-from the page they are visiting to another page in in the wiki. In the
-`English Wikipedia`__, redirects make up more than half of all article
-pages.
+In wikis, **redirects** are special pages in that silently take
+readers from the page they are visiting to another page in the
+wiki. In the `English Wikipedia`__, redirects make up more than half
+of all article pages.

 
 

-Different data sources of handle redirects differently. For example,
-`the MediaWiki API`__ will automatically "follow" redirects but the `XML
-database dumps`__ treat redirects like normal articles. In both cases,
-redirects are often invisible to reseachers.
+Different Wikipedia data sources handle redirects differently. For
+example, `the MediaWiki API`__ will automatically "follow" redirects
+but the `XML database dumps`__ treat redirects like normal
+articles. In both cases, redirects are often invisible to researchers.

 
 __ https://www.mediawiki.org/wiki/API:Main_page
 __ https://meta.wikimedia.org/wiki/Data_dumps
 
 Because redirects constitute a majority of all pages and see a large
 
 __ https://www.mediawiki.org/wiki/API:Main_page
 __ https://meta.wikimedia.org/wiki/Data_dumps
 
 Because redirects constitute a majority of all pages and see a large

-portion of all traffic, Wikipedia reseachers need to take redirects into
-account or their findings may be incomplete or incorrect.  For example,
-the histogram on this page shows the distribution fo edits across pages
-in Wikipedia for every page, and for non-redirects only.  Because
-redirects are almost never edited, the distributions are very different.
-Similarly, because redirects are viewed but almost never edited, any
-study of views over articles should also take redirects into account.
+portion of all traffic, Wikipedia researchers need to take redirects
+into account or their findings may be incomplete or incorrect.  For
+example, the histogram on this page shows the distribution of edits
+across pages in Wikipedia for every page, and for non-redirects only.
+Because redirects are almost never edited, the distributions are
+very different.  Similarly, because redirects are viewed but almost
+never edited, any study of views over articles should also take
+redirects into account.

 
 .. figure:: edits_over_pages.png
    :align: right
 
 .. figure:: edits_over_pages.png
    :align: right


@@ -54,7 +55,7 @@ when and where pages redirect.
 
 Much more detail can be found in `Consider the Redirect: A Missing
 Dimension of Wikipedia Research`__ — a short paper that we have written
 
 Much more detail can be found in `Consider the Redirect: A Missing
 Dimension of Wikipedia Research`__ — a short paper that we have written

-to acccompany this dataset and these tools.  If you use this software or
+to accompany this dataset and these tools.  If you use this software or

 these data, we would appreciate if you cite the paper:
 
   *Hill, Benjamin Mako and Aaron Shaw. "Consider the Redirect:  A Missing
 these data, we would appreciate if you cite the paper:
 
   *Hill, Benjamin Mako and Aaron Shaw. "Consider the Redirect:  A Missing


@@ -78,25 +79,25 @@ Generating redirect spells from an MediaWiki XML dump involves two steps:
 We have `publicly released software in Python and R to do these two
 steps`__ under the `GNU GPL version 3`__. The software is designed for
 people already comfortable with working with MediaWiki XML dumps and the
 We have `publicly released software in Python and R to do these two
 steps`__ under the `GNU GPL version 3`__. The software is designed for
 people already comfortable with working with MediaWiki XML dumps and the

-tools and software necessary to do this. We have provided
-`documentation`__ on how to use these tools.
+tools and software necessary to do this.

 
 __ http://projects.mako.cc/source/?p=redirect-tools
 __ http://www.gnu.org/licenses/gpl-3.0.html
 
 __ http://projects.mako.cc/source/?p=redirect-tools
 __ http://www.gnu.org/licenses/gpl-3.0.html

-__ README.html

 
 You can download the software from our git repository like::
 
   git clone git://projects.mako.cc/redirect-tools
 
 
 You can download the software from our git repository like::
 
   git clone git://projects.mako.cc/redirect-tools
 

-Detailed documentation on how to use the software is in available in our
-README file.
+Detailed documentation on how to use the software is in available in `our
+README file`__.
+
+__ README.html

 
 Redirect Spell Data
 =========================
 
 
 Redirect Spell Data
 =========================
 

-In our paper `Consider the Redirect`__, we present an analysis of
-redirect data from English Wikipedia in the dump created in October
+In `our paper`__, we present an analysis of redirect data from English
+Wikipedia in the dump created in October

 2012. You can download `the dump files we used`__ from `the Wikimedia
 Foundation dataset archive`__.  Because generating these dumps can be
 computationally intense, we have published the output of the software
 2012. You can download `the dump files we used`__ from `the Wikimedia
 Foundation dataset archive`__.  Because generating these dumps can be
 computationally intense, we have published the output of the software


@@ -105,14 +106,15 @@ our software identified and is the dataset used in the paper.
 
 You can download the dataset in the following formats:
 
 
 You can download the dataset in the following formats:
 

-- `RData (240MB)`__ — Suitable for use in GNU R
-- `bzip2 compressed tab seperated values (178MB)`__ — Suitable for use
+- `RData (240MB)`__ — Suitable for use in `GNU R`__
+- `bzip2 compressed tab separated values (178MB)`__ — Suitable for use

   in other languages and statistical packages.
 
 __ http://mako.cc/academic/hill_shaw-consider_the_redirect.pdf
 __ http://dumps.wikimedia.org/enwiki/20121001/
 __ http://dumps.wikimedia.org/
 __ enwiki_201210-redirect_spells-v1.RData
   in other languages and statistical packages.
 
 __ http://mako.cc/academic/hill_shaw-consider_the_redirect.pdf
 __ http://dumps.wikimedia.org/enwiki/20121001/
 __ http://dumps.wikimedia.org/
 __ enwiki_201210-redirect_spells-v1.RData

+__ http://www.r-project.org/

 __ enwiki_201210-redirect_spells-v1.tsv.bz2
 
 More Information
 __ enwiki_201210-redirect_spells-v1.tsv.bz2
 
 More Information


@@ -137,3 +139,10 @@ a patch using git are online`__.
 __ http://projects.mako.cc/source/
 
 
 __ http://projects.mako.cc/source/
 
 

+----
+
+ⓒ Copyright `Benjamin Mako Hill`__ and `Aaron Shaw`__ :: `Creative Commons BY-SA`__ :: Updated: Thu Jul  3 13:22:29 PDT 2014
+
+__ http://mako.cc/academic/
+__ http://aaronshaw.org/
+__ http://creativecommons.org/licenses/by-sa/4.0/