Appropedia:Admin tasks/Interwiki map

[edit] Initial Population

[edit] Meatball Wiki

MediaWiki looks to Meatball Wiki for the basic, current Interwiki mapping list. Find the current info here, http://www.usemod.com/cgi-bin/mb.pl?InterMapTxt Download the text file here, http://usemod.com/intermap.txt Use the format conversion command found below and past into the Interwiki map edit page.

[edit] Included by Mediawiki

Additionally, Mediawiki includes a couple files in the maintenance folder. However, they contain only the short prefixes. Also, they need to be edited slightly before calling them into the mySQL interwiki table.

[edit] Editing Included Interwiki files

Everything above the line

REPLACE INTO /*$wgDBprefix*/interwiki (iw_prefix,iw_url,iw_local) VALUES

needs to be deleted. And our own interwiki table name needs to replace theirs, making it,

REPLACE INTO /*$wgDBprefix*/w1interwiki (iw_prefix,iw_url,iw_local) VALUES

The files Mediawiki includes are:

interwiki.sql

wikipedia-interwiki.sql

wiktionary-interwiki.sql

If any of these files are included in the script oulined below, you have to re edit them whenever a Mediawiki Version Upgrade is conducted, because they get overwritten in the process.

I have only included wikipedia-interwiki.sql into the script because it contains all the links to wikipedia's foreign language pages. I didn't include interwiki.sql because the Meatball version is better. And, I didn't include the wiktionary-interwiki.sql because it's contents seemed unnecessary at this time. We may want to include it into the script in the future. You can do so by adding this line to the end of the script

mysql -u**** -p**** -h mysql.whatissustainability.org appropedia </home/lonny1/appropedia.org/maintenance/wiktionary-interwiki.sql;

just under the line to add wikipedia-interwiki.sql.

[edit] Update Script

This script updates the Interwiki tables in our database by parsing the most current tables posted at Interwiki map.

The script lives here:

~/appropedia.org/maintenance/Interwiki/UpdateInterwiki.sh

Its contents:

#!/bin/bash
# Grab interwiki map from website
lynx -dump -nolist -width=160 http://appropedia.org/Interwiki_map | gawk '/ http:/ { print $1 "\t" $2 }' >~/appropedia.org/maintenance/Interwiki/interwikitable.txt
# Run mySQL batch script using batchfile TableUpload.sql. An empty tableupload.log should mean no errors.
mysql -u**** -p**** -h mysql.whatissustainability.org appropedia </home/lonny1/appropedia.org/maintenance/Interwiki/TableUpload.sql >tableupload.log;
# Run mediawiki included mySQL batch scripts for the standard Wikimedia Interwiki prefixes
# Make sure these have been modified to our database naming scheme and have had the header notes removed
mysql -u**** -p**** -h mysql.whatissustainability.org appropedia </home/lonny1/appropedia.org/maintenance/wikipedia-interwiki.sql;

[edit] mySQL Subscript

The script lives here:

~/appropedia.org/maintenance/Interwiki/UpdateInterwiki.sh

Its Contents:

DELETE FROM w1interwiki;
LOAD DATA LOCAL INFILE '/home/lonny1/appropedia.org/maintenance/Interwiki/interwikitable.txt' INTO TABLE w1interwiki;

[edit] crontab

Currently this script is run daily to keep our interwiki table current. It's excessive because it's unlikely that our Interwiki table is being changed that often. A more ideal solution would be to have a page-change hook script activate this update script. Until then, here's the crontab command

/home/lonny1/appropedia.org/maintenance/Interwiki/UpdateInterwiki.sh

Make sure to add the desired frequency info to the front.

[edit] Table Format Conversions

All commands are written for the unix environment

[edit] White Space to Tab Delimited

Population of the Interwiki table requres specific formatting, called out in the mysql command, LOAD DATA INTO TABLE. The script used for appropedia.org uses mysql's default format, tab delimits per line of entry.

gawk '/ http:/ { print $1 "\t" $2 }' file.in >file.out

This command line requres that each Interwiki table entry is on its own line and that the prefix is seperated by its address by some amount of white space. Also, that the prefix is the first entry and the address is the second. This format is how Meatball Wiki provides their InterMap textfile.

[edit] White Space to Wiki Code

A quick way to write an Interwiki table in Wiki code, correctly formatted for our script to parse.

gawk 'BEGIN { print "\{| class\=\"plainlinks\"" } ; { print "| " $1 " || " $2 "\n|-" } ; END { print "|\}" }' file.in >file.out

This command requres input similar to what is produced by the command 'White Space to Tab Delimited' above. Similar meaning, tabs and/or multiple spaces (any white space).