You can have your own OTT mirror now!

If you want to run your own copy of the ЯOЯЯIM TTO, you can, because I made it available. Follow these instructions to download and setup your own ЯOЯЯIM TTO.

This is about version 1, iteration 1.

Dependencies

The mirror depends on some things:

• OS:
  The mirror was written and tested on Debian, and then Cubian. Other versions of GNU/Linux should be okay.
• a www server:
  I use apache2. Any other server that supports CGI should be okay. The web server must support URLs without extension in filename when serving archived attachments and images
• AWK interpreter: mawk
  gawk should be compatible, so you can use it too.
• C++ compiler: g++
  The mirror is written in C++, but doesn't use anything C++-specific. I may convert it to C in the future.
• CGI library: cgilib
  There is a bug in cgilib. At least the version from the Debian repository. Change line 146 of cgi.h from "extern }" to "}"
• wget
• Something to schedlue the bots
  The bots aren't daemons and need something to start them in regular time intervals. I use cron.

Downloading

• Create a directory where the mirror will be hosted. Download ott.zip and put its contents there. Remember that the mirror must be hosted at "something.example.com/ott"
• Create a directory for the source. Download pro.zip and put its contents there. Don't build it yet.

This and earlier versions can also be downloaded from svn://bicyclesonthemoon.info/ottmirror. Use "e" for read-only access.
If you want the source as it is right now, get it from http://1190.bicyclesonthemoon.info/ott/src/.

Preparing the enviroment

Go to where you downloaded the source. Open re.awk. At hte top of the file there are some paths defined. You'll have to change some of these. They have to be double-escaped. The directories should not end with a "/".

• Set wgetpath to point to wget. Set mawkpath to point to mawk.
• Create a directory where the programs will be run from. Can be the same where you downloaded the source. Set propath to point there.
• Create a directory for temporary files. Only you should have read access there. While someone makes a post, their password can be found in a temporary file. Set tmppath to point there.
• Create a directory where the bots will remember things. Set mempath to point there. Create a file named "name" there. Put your URL-encoded xkcd fora login in the first line and your URL-encoded password in the second line. Only you should have read access to the file.
  Create the following subdirectories:
  • mlist
  • mpost
  • mpost/fail
  • mpost/ok
  • mpost/pm
  Only you should have read access to mpost. Passwords can be found there.
• Create a directory where the bots will write their reports. set logpath to point there.
• Set mirrpath to point at the directory where the mirror is hosted. That's where you put the contents of ott.zip
• Set your useragents. useragent2 is used by bothasar_t, the mirror bot. useragent3 is used by bothasar_p, the post bot.
• Set own_image_regexp to match the URL where the images are archived. The URL should be something like "something.example.com/ott/image/".
• Set own_url_encoded to your URL-encoded mirror URL.
• Now save the file and build the programs using the makefile.
• Copy the binaries and AWK scripts to where propath points to. Unless it's the same directory.
• Go to where the mirrpath points to. You may want to edit the following files:
  • np/top - the newpage top.
  • np/end - the newpage bottom.
  • top - the indexpage top.
  • end - the indexpage bottom.
  • post.htm - the post page.
  • info.htm - the information page.
  • update.htm - the update page.
  so that you can set your own title image, links, etc.
  Some lines look like ###this. Don't change them. The programs will place their content there.

Setting up the server

You have to set up server so that some URLs will link to some CGI programs.

• "/ott/log" should link to "logpath".
• "/ott/view" should link to "propath/view".
• "/ott/mview" should link to "propath/mview".
• "/ott/update" should link to "propath/update".
• "/ott/index" should link to "propath/index".
• "/ott/post" should link to "propath/post".
• "/ott" should link to "propath/index".

That's how I did it in apache2:

Alias /ott/log /eizm/log/ottmirror
<Directory "/eizm/log/ottmirror">
 Options Indexes
 Order allow,deny
 Allow from all
</Directory>

ScriptAlias /ott/view /eizm/pro/ottmirror/view
ScriptAlias /ott/mview /eizm/pro/ottmirror/mview
ScriptAlias /ott/update /eizm/pro/ottmirror/update
ScriptAlias /ott/index /eizm/pro/ottmirror/index
ScriptAlias /ott/post /eizm/pro/ottmirror/post
ScriptAliasMatch ^/ott/?$ /eizm/pro/ottmirror/index


Commandline parameters for bot2

bot2 (bothasar_t) is the thread archiving bot.

• parameters with a numeric value:
  • -i - the bot ID
    There can be multiple copies of the bot. Each should have its own ID. They have their own tempfiles, their own logs, and their own memory of what was the last newpage. The bot will refuse to start when there is another bot with the same ID still running. The default value is 0. Don't use the value 4 because it's already used in update.1.cpp.
  • -o - offset from the prievious page
    It determines where to start relative to the last page saved on the previous run (of the same ID). 0 means downloading the next page, 1 the same page, 2 the previous page, 3 the previousprevious page, and so on. If this is the first run, it will start from the first page. The default value is 1.
  • -m - maximal number of pages in one run
    The bot will not download more pages than this. The default value is 1.
  • -p - distance from the last page
    The bot will always stay at least this many pages away from the latest page on the thread. Default value is 0.
  • -s - start position override
    The bot will start from this page regardless of where it stopped last time
  • -w - wait between pages
    Time in seconds to wait between downloading pages. The default value is 3.
  • -v - wait after download
    Time in seconds to wait after downloading an avatar, attachment or image. The default value is 15.
• parameters without a value:
  • -d - download avatars
    If this option is set, the bot will download avatars.
  • -a - download attachments
    If this option is set, the bot will download attachments.
  • -b - download images
    If this option is set, the bot will download images that aren't avatars or attachments.
  • -t - stdout
    If this option is set, the bot will write to standard output instead of the log file.
  • -n - new log
    If this option is set, the bot will replace the old log. Otherwise it will add to it.
  • -r - stay in present
    If this option is set, after reaching the last page the bot will stay there and will also download the index. Otherwise it will continue from the first page.

Commandline parameters for bot3

bot3 (bothasar_p) is the post bot.

• parameters with a numeric value:
  • -w - wait between posts
    Time in seconds to wait between sending posts. Or between a failed post and sending a PM. The default value is 15.
• parameters without a value:
  • -t - stdout
    If this option is set, the bot will write to standard output instead of the log file.
  • -n - new log
    If this option is set, the bot will replace the old log. Otherwise it will add to it.

Scheduling the bots

The bots won't run on themselves. They need something that will start them at regular time intervals. I will share my crontab configuration.

20,50 * * * * /eizm/pro/ottmirror/bot2 -i2 -r -o1 -m5 -w9 -v5 -d -a
This bot runs every 30 minutes and updates the pages with new posts.

26 23 * * * /eizm/pro/ottmirror/bot2 -i3 -r -o5 -m10 -p2 -w9 -v5 -d -a -b
This bots runs every day and reloads the pages with new posts and 4 previous pages except the two last pages. Because there could be a delurker or someone could make edits which we don't want to miss.

23 * * * * /eizm/pro/ottmirror/bot2 -i1 -o0 -m3 -p2 -w9 -v5 -d -a -b
This bot runs every hour and goes slowly through the whole thread except the two latest pages. Because there can be temporal edits. (like the 1300 and 1800 repositories).

While the thread being catched up for the first time I'd recommend running only one copy of the bot at a faster rate than this.

7,27,47 * * * * /eizm/pro/ottmirror/bot3 -w15
This bot runs every 20 minutes and looks for new posts to be sent to the thread.

0 0 * * 1 /bin/mv /eizm/log/ottmirror/bot2.log.1 /eizm/log/ottmirror/bot2.log.1.lastweek
0 0 * * 1 /bin/mv /eizm/log/ottmirror/bot2.log.2 /eizm/log/ottmirror/bot2.log.2.lastweek
0 0 * * 1 /bin/mv /eizm/log/ottmirror/bot2.log.3 /eizm/log/ottmirror/bot2.log.3.lastweek
0 0 * * 1 /bin/mv /eizm/log/ottmirror/bot3.log /eizm/log/ottmirror/bot3.log.lastweek
This moves the log files once in a week. Otherwise they would grow to infinity.

Changes

• v.1, i.1 (12.08.2014):
  • Added AGPL license.
  • Added BBcode to HTML translation.
  • Fixed a bug causing bot2 to stop on 403 error when downloading attachments.
• v.1, i.0 (17.07.2014):
  • First publicly available version

Copyright, license, etc.

The code (in pro.zip) is written by me, Balthasar Szczepański and is released under the AGPL 3license since version 1, iteration 0. v.1, i.0 doesn't have any license defined.

The resources in ott.zip contain HTML, CSS, Javascript and images from phpBB which is released under the GPL 2 license.

Congratulations

If you managed to do everything described here you should have your own ЯOЯЯIM TTO. Now wait for it until it catches the whole thread.

(Unless I forgot about something important here. Let me know if I did)

Balthasar