Comments on: Installing Wikipedia part 4 of N: getting additional wikipedia metadata

By: arunxjacob

arunxjacob — Thu, 19 Jun 2008 04:40:40 +0000

I was more interested in the raw schema than the rendered contents, which is why I didn’t go the wikimedia route. Pagelinks shows the inter page links, which correlates to the relevancy of the page being linked to.

By: wantondevious

wantondevious — Fri, 13 Jun 2008 00:08:25 +0000

Hi,

I hadn’t found this before I started this project myself. However, the approach I took was to install Wikimedia, which in turn creates the database (plus allows you to render the dump). Currently Im at 4.7 mill pages after 1 day – Im cursing that I didnt turn off Binary Logging and DISABLE the page and text indexes.

I’m not sure whether this would make it faster, but turning off indexes usually does.
In general binary log is ok in MySQL (doc claims only 1% overhead – but… we’re loading 12-13 GB of text – and so this creates 12-13 GB worth of logs.

My question is – what does the pagelinks actually DO? MediaWiki seems to render everything just fine, without this extraction. How would it generate this table in normal usage? Or is it an adhoc table thats only populates on demand?

I’d like to have this data, but if mediawiki will generate for me, seems silly to upload the pagelink table.

By: arunxjacob

arunxjacob — Sun, 11 May 2008 06:06:39 +0000

That’s strange, we actually had issues getting it to run under postgres. I wasn’t the guy trying to do this, but he was bummed he couldn’t get it working on ‘a real database instead of that toy MySQL database!’ (his words, not mine, I’m not religious about this stuff

By: geoffsyndicate

geoffsyndicate — Sun, 04 May 2008 04:58:42 +0000

Although I found this series of posts to be very useful for first-time Wikipedia importers (thank you for posting this BTW), I eventually gave up on MySQL. It turns out that MySQL was crashing pretty severely when I tried to import the categorylinks data, so I have written a conversion script and imported the whole lot into PostgreSQL (without any problems).

There you go. MySQL == bad, PostgreSQL == good.

By: geoffsyndicate

geoffsyndicate — Thu, 01 May 2008 22:43:29 +0000

I’m on a Macbook Pro, 2.4Ghz with 4 gigs of RAM. I’m thinking about cancelling it and removing the locks on the sql so I can monitor it’s progress. Thoughts?

By: arunxjacob

arunxjacob — Thu, 01 May 2008 21:32:01 +0000

It didnt take more than an hour, what kind of hardware are you running on? I was running this on a quad core 64 bit server.

By: geoffsyndicate

geoffsyndicate — Thu, 01 May 2008 21:23:00 +0000

Hi,

How long did it take to load the category links and page links? The category links script locks the table, so I don’t know of any way to tell how far through it is. It’s been going for a while now…

By: indigene

indigene — Mon, 21 Apr 2008 13:27:29 +0000

Arun

Read with great interest your travails to get the local Wikipedia up and running. Phew!

I am building a web application using the mediawiki platform on LAMP. Currently doing the DB design with MySQL Work Bench.

I have imported the mediawiki schema you mention in your article:

http://www.mediawiki.org/wiki/Manual:Database_layout

using this schema xml file:
http://files.nickj.org/MediaWiki/mediawiki-dbdesigner-schema-data.xml)

which also generated this cute layout pic:

http://upload.wikimedia.org/wikipedia/commons/4/41/Mediawiki-database-schema.png

But would love to look at an actual layout of the wikipedia db. Can you generate a db schema file from your local Wikipedia and send me in a format (sql/xml) that I can feed into Work Bench?

I am indigene2007 on gmail

Thanks in advance,
Warm Regards