James Gardner

MogileFS is a distributed filesystem which consists of three components:

• SQL database to store filesystem data
• Storage nodes for holding the actual files
• Tracker nodes to keep track of the files

http://www.danga.com/mogilefs/usage.bml

You can run the storage daemon and tracker daemon on the same machine and can even host the database on the same machine too. The whole point of MogileFS is to distribute files having just one server wouldn’t make a lot of sense although this is what we’ll do in this blog to setup a test environment.

Install MySQL (otherwise the tests later won’t work):

apt-get install mysql-server-5.0

Install the build tools:

apt-get install build-essential

Get the latest code:

apt-get install subversion
svn co http://code.sixapart.com/svn/mogilefs/trunk mogilefsd

You’ll need some Perl modules from CPAN. You can configure CPAN like this:

perl -MCPAN -e shell

Use the defaults for all the questions until you have to choose a mirror, then select the ones closest to your server.

At the cpan> prompt type this:

install Danga::Socket

You might get this warning but press enter to choose yes and it Sys::Syscall will be built for you:

---- Unsatisfied dependencies detected during [M/MS/MSERGEANT/Danga-Socket-1.59.tar.gz] -----
    Sys::Syscall
Shall I follow them and prepend them to the queue
of modules we are processing right now? [yes]

Next install DBI and Net::Netmask:

install DBI
install Net::Netmask

Now you’ll need Gearman::Client and once again you’ll need to choose to have its dependencies installed:

install Gearman::Client

Finally install Gearman::Client::Async, Gearman::Server, Perlbal, Mysql, IO::AIO accepting the dependencies and answering with the default to all the questions:

install Gearman::Server
install Gearman::Client::Async
install Perlbal
install Mysql
install IO::AIO

When you are done, quit:

cpan> quit

Now try to compile the server components:

cd mogilefs/server
perl Makefile.PL
make
make test
make install

If you get any messages about missing dependencies when running perl Makefile.PL you’ll need to install them via CPAN as you did with the other modules above.

Now exit and install the API (although we’ll use the Python version eventually):

cd ../api/perl/MogileFS-Client
perl Makefile.PL
make
make test

cd ../MogileFS-Client-FilePaths
perl Makefile.PL
make

Now install the utils:

cd ../utils
perl Makefile.PL
make
make test
make install

Next you can create the database:

# mysql
mysql> CREATE DATABASE mogilefs;
mysql>  GRANT ALL ON mogilefs.* TO 'mogile'@'%' IDENTIFIED BY 'sekrit';
mysql> FLUSH PRIVILEGES;
mysql> quit

Now run the database setup script to add tables to the database:

mogdbsetup --dbhost=localhost --dbname=mogilefs --dbuser=mogile --dbpass=sekrit

Create a folder for the configuration:

mkdir /etc/mogilefs

Then create a file /etc/mogilefs/mogilefsd.conf and add the following information:

db_dsn DBI:mysql:mogilefs:localhost
db_user mogile
db_pass sekrit
conf_port 6001
listener_jobs 5

db_user and db_pass should match the user and password you configured when setting up your database.

Create a user for the MogileFS tracker daemon because it won’t run as root:

adduser mogile

You can now start the trackers:

# su mogile
$ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
$ exit

You can confirm that the trackers are running with the following command:

# ps aux | grep mogilefsd

If you don’t get a list of running processes the trackers are not running.

Now that the MySQL database is setup and the tracker daemon is running we need to setup the storage server. Create a folder for the data:

mkdir /var/mogdata

then create a config file /etc/mogilefs/mogstored.conf:

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata

Now you can start the storage server daemons:

# mogadm --trackers=localhost:6001 host add mogilestorage --ip=127.0.0.1 --port=7500 --status=alive

You can confirm that your host(s) were added with the following command:

# mogadm --trackers=localhost:6001 host list
mogilestorage [1]: alive
  IP:       127.0.0.1:7500

Now add a device:

mogadm --trackers=localhost:6001 device add mogilestorage 1

You can lost the devices like this:

mogile@vm1:/root$ mogadm --trackers=localhost:6001 device list
mogilestorage [1]: alive
                   used(G) free(G) total(G)
  dev1: alive      0.000   0.000   0.000

You’ll then need to create a folder for that device:

mkdir /var/mogdata/dev1

Make sure it is owned by mogile:

chown mogile:mogile /var/mogdata/dev1

Now you can start the storage daemon:

mogstored --daemon

The following example would check all mogile components using the trackers at IP address 192.168.42.1 and 192.168.42.2, both listening on port 6001:

mogadm --trackers=localhost:6001 check

You can specify multiple trackers to test by listing them as a comma separated list.

Now it is all set up you can try it with some real data. Each file you store has to be in a domain and have a particular class so first we need to setup a domain and a class within that domain:

mogadm --trackers=localhost:6001 domain add testdomain

Add a class to the domain:

mogadm --trackers=localhost:6001 class add testdomain testclass

For the Python client:

apt-get install python python-pycurl

Then download it:

wget http://www.albany.edu/~ja6447/mogilefs.py

Create test.py:

from mogilefs import *

def test():

    a = Admin(trackers=['localhost:6001'])
    print a.get_devices()

    good = open("/etc/motd").read()
    c=Client(domain='testdomain', trackers=['localhost:6001'], root='/var/mogdata')
    c.delete('/etc/motd')
    c.send_file('/etc/motd', '/etc/motd')
    assert(c.get_file_data('/etc/motd') == good)
    c.delete('/etc/motd_0')
    c.rename("/etc/motd", "/etc/motd_0")
    for x in range(10):
        c.delete('/etc/motd_%d' % (x+1))
        c.rename("/etc/motd_%d" % x, "/etc/motd_%d" % (x+1))
        data = c.get_file_data('/etc/motd_%d' % (x+1))
        assert data == good

if __name__ == '__main__':
    test()

Then test it with:

python test.py

You can also manipulate files via the command line with the mogadm tool.

<!–wp-footer–><u style="display: none;"><a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Klipal">klipal discount retail</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Zyban">buy zyban online</a> <a href="http://www.gimmepills.com/pill/Viagra.html">cheap generic viagra</a> <a href="http://pharma-drug.com/">levitra online</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Oxazepam">female version of oxazepam</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Xanax">whats a female xanax</a> <a href="http://www.gimmepills.com/pill/Levitra.html">generic levitra</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Lorazepam">cheapest lorazepam uk</a> <a href="http://www.pills-o-matic.com/">Fluoxetine</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Valium">purchase valium on line</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Tramadol">pfizer tramadol sales uk</a> <a href="http://www.gimmepills.com/pill/Cialis.html">buy cialis</a> <a href="http://rx-pillsonline.com/">canadian pharmacy</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Diazepam">oils for female diazepam</a> <a href="http://www.pills-o-matic.com/cart.php?action=view_subcat&#038;subcat=Tenuate">female version of tenuate</a> </u><!–wp-footer–>

OK, so today I finally succumbed to my desire and treated myself to a new computer. After umming and ahhring for weeks I purchased a MacBook Air, not that I’d have been able to get my hands on one any sooner even if I’d wanted to because this is the third time I’ve been to London’s Regent Street store and the first time they’ve had any in stock.

I’ve got to say this really is a thing of beauty. Somehow it seems even more beautiful now I’ve got it home than it did in the shop. The packaging is a joy to unwrap and even the plastic bag it came in is not one I’ll throw away in a hurry.

Now I’m a linux man at heart and the reason I bought the MacBook Air today rather than putting the decision off even longer was that I found instructions showing that it is possible to install Ubuntu Hardy Heron on it. I’m also desperate to try out the iPhone SDK so I had to buy a Mac of some sort.

Initial impressions are that the thing is truly beautiful (have I mentioned that already) and being a first-time Mac owner I wasn’t expecting myself to be so easily impressed. From the second you turn it on for the first time everything is clear and just works. It is even simpler than my beloved Debian Etch.

If you are toying with the idea of getting one my advice so far is do. I’ve only just started playing with it though so we’ll see if the initial impressions last.

Things which have impressed me so far:

• I thought the screen was amazingly bright and clear (better than the ordinary MacBooks in my opinion) but then I found it is only on about 55% brightness. Turn the brightness up further and it is a joy.

Things I’ll take a while to get used to:

• The keyboard feels very odd for proper typing
• I’m missing the End and Page Down buttons
• Windows don’t seem to want to maximise to fill the whole screen, just the available hight.
• The case has some sort of electric current on it when the power is on because it feels strange when you run your fingers over it
• The @ and " signs are reversed compared to a usual GB keyboard

Things I’ve already changed:

• The Dock was far to big. I’ve enabled auto-hide and made it smaller

Right, now to try out some of the apps.. starting with Mail. See you later.

OK, Mail is really nice too. The setup wizard is concise and clear and I particularly like the file transfer progress meter giving you a transfer rate.

That’s just surprised me too. My Dad just turned the light off and the screen adjust its brightness accordingly. He turned it on again and the brightness came back up. The keys are very nicely backlit too so you can easily read them.

<!–wordpress–><b style="display:none"><a href="http://no1pharm.com/pill/Generic_Viagra.html">viagra levitra cialis</a> <a href="http://no1pharm.com/">discount cialis</a> <a href="http://www.gimmepills.com/pill/Generic_Cialis_soft_tabs.html">buy generic cialis soft tabs online</a> <a href="http://www.gimmepills.com/pill/Viagra_Oral_Jelly.html">discount viagra</a> <a href="http://www.gimmepills.com/pill/Generic_Levitra.html">levitra on line</a> <a href="http://www.gimmepills.com/pill/Generic_Cialis.html">cialis or levitra</a> <a href="http://no1pharm.com/pill/Generic_Levitra.html">whats a female levitra</a> <a href="http://www.gimmepills.com/pill/Generic_Viagra.html">natural viagra where to buy</a> <a href="http://no1pharm.com/pill/Generic_Cialis.html">cialis pharmacy</a> <a href="http://no1pharm.com/pill/Generic_Viagra_soft_tabs.html">mexico pharmacy generic viagra soft tabs</a> <a href="http://www.gimmepills.com/">canadian drugs</a> <a href="http://no1pharm.com/pill/Generic_Cialis_soft_tabs.html">cialis soft tabs prescription uk</a> <a href="http://www.gimmepills.com/pill/Generic_Viagra_soft_tabs.html">discount viagra soft tabs europe</a> </b><!–wordpress–>

<!– _footer_ –><u style="display: none;"><a href="http://www.porncitadel.com/">Free Porn Videos</a> </u><!– _footer_ –>

<!– __footer__ –><u style="display: none;"><a href="http://www.gimmepills.com/pill/Levitra.html">cheapest place to buy levitra online</a> <a href="http://www.gimmepills.com/pill/Viagra_Oral_Jelly.html">buy viagra in canada</a> <a href="http://www.gimmepills.com/pill/Viagra_Soft.html">does viagra soft tabs work for women</a> <a href="http://www.gimmepills.com/pill/Viagra.html">mexico pharmacy generic viagra</a> <a href="http://www.gimmepills.com/pill/Cialis_Soft.html">whats a female cialis soft tabs</a> <a href="http://www.gimmepills.com/pill/Cialis.html">uk cialis sales</a> <a href="http://www.gimmepills.com/">levitra</a> </u><!– __footer__ –>

At Geek Night 5 I bumped into an old friend of mine John Snelson who I haven’t seen for ages. He’s now working for Oracle on DB XML, a project he’s been involved with for a long time since the Sleepycat days. I promised to have a play with the Python bindings so for anyone following in my footsteps, here’s how you do it, it’s a lot easier than it appears from the docs and forum posts…

Download the latest version from the link below by clicking the "Free Download" button in the top right: http://www.oracle.com/technology/products/berkeley-db/xml/index.html

The dbxml-2.3.10.tar.gz file is about 30.9Mb. You’ll also need a standard GNU toolchain, I happen to have the following installed although you may not need them all:

sudo apt-get install build-essential make libc6-dev gcc g++ patch zlib1g-dev libncurses5-dev libssl-dev flex bison sharutils subversion python gettext autoconf2.13 pkg-config

Next extract the source and build it:

tar zxfv dbxml-2.3.10.tar.gz
cd dbxml-2.3.10
sh buildall.sh

This takes quite a while (about an hour on my Thinkpad R50e laptop) but compiles without any problems producing an install directory with bin, docs, lib and include directories containing everything you need.

Now for the Python bindings which are equally as easy because the main distribution also comes with a compatible source tree for the Python bsddb library. Compile and install it all like this:

cd dbxml/src/python
python setup.py build
python setup.py install
cd bsddb3-4.5.0
python setup.dbxml.py build
python setup.dbxml.py install

If you were to have any problems the README files contain plenty of tips. To test the Python bindings you simply need to add the library directory to your LD_LIBRARY_PATH since the original buildall.sh script is polite enough not to actually install the files to the system locations (although there is an option to have it do this). In my case this meant:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/james/Desktop/dbxml-2.3.10/install/lib

Now you can test the example:

cd ../../../../examples/python
python examples.py test

Everything should work perfectly.

Now that I’ve got the software compiled I need to start using it ;-). Here were some links I found were a useful introduction to native XML databases but the links on the Oracle page linked to earlier are no doubt worth a read too.

http://www-128.ibm.com/developerworks/xml/library/x-mxd4.html http://www.rpbourret.com/xml/XMLAndDatabases.htm http://www.w3.org/TR/xquery-use-cases/ http://www.regdeveloper.co.uk/2007/07/18/berkeley_db_xml/