MogileFS on Debian Etch ++++++++++++++++++++++++++ :Posted: 2008-04-10 00:40 :Tags: XML .. caution :: Although I've got MogileFS working there was a bit of trial and error involved. I believe what I've documented here is the path I have taken but I haven't double checked it. MogileFS is a distributed filesystem which consists of three components: * SQL database to store filesystem data * Storage nodes for holding the actual files * Tracker nodes to keep track of the files http://www.danga.com/mogilefs/usage.bml You can run the storage daemon and tracker daemon on the same machine and can even host the database on the same machine too. The whole point of MogileFS is to distribute files having just one server wouldn't make a lot of sense although this is what we'll do in this blog to setup a test environment. Install MySQL (otherwise the tests later won't work):: apt-get install mysql-server-5.0 Install the build tools:: apt-get install build-essential Get the latest code:: apt-get install subversion svn co http://code.sixapart.com/svn/mogilefs/trunk mogilefsd You'll need some Perl modules from CPAN. You can configure CPAN like this:: perl -MCPAN -e shell Use the defaults for all the questions until you have to choose a mirror, then select the ones closest to your server. At the ``cpan>`` prompt type this:: install Danga::Socket You might get this warning but press enter to choose ``yes`` and it ``Sys::Syscall`` will be built for you:: ---- Unsatisfied dependencies detected during [M/MS/MSERGEANT/Danga-Socket-1.59.tar.gz] ----- Sys::Syscall Shall I follow them and prepend them to the queue of modules we are processing right now? [yes] Next install ``DBI`` and ``Net::Netmask``:: install DBI install Net::Netmask Now you'll need ``Gearman::Client`` and once again you'll need to choose to have its dependencies installed:: install Gearman::Client Finally install ``Gearman::Client::Async``, ``Gearman::Server``, ``Perlbal``, ``Mysql``, ``IO::AIO`` accepting the dependencies and answering with the default to all the questions:: install Gearman::Server install Gearman::Client::Async install Perlbal install Mysql install IO::AIO When you are done, quit:: cpan> quit Now try to compile the server components:: cd mogilefs/server perl Makefile.PL make make test make install If you get any messages about missing dependencies when running ``perl Makefile.PL`` you'll need to install them via CPAN as you did with the other modules above. Now exit and install the API (although we'll use the Python version eventually):: cd ../api/perl/MogileFS-Client perl Makefile.PL make make test cd ../MogileFS-Client-FilePaths perl Makefile.PL make Now install the utils:: cd ../utils perl Makefile.PL make make test make install Next you can create the database:: # mysql mysql> CREATE DATABASE mogilefs; mysql> GRANT ALL ON mogilefs.* TO 'mogile'@'%' IDENTIFIED BY 'sekrit'; mysql> FLUSH PRIVILEGES; mysql> quit Now run the database setup script to add tables to the database:: mogdbsetup --dbhost=localhost --dbname=mogilefs --dbuser=mogile --dbpass=sekrit Create a folder for the configuration:: mkdir /etc/mogilefs Then create a file ``/etc/mogilefs/mogilefsd.conf`` and add the following information:: db_dsn DBI:mysql:mogilefs:localhost db_user mogile db_pass sekrit conf_port 6001 listener_jobs 5 ``db_user`` and ``db_pass`` should match the user and password you configured when setting up your database. Create a user for the MogileFS tracker daemon because it won't run as root:: adduser mogile You can now start the trackers:: # su mogile $ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon $ exit You can confirm that the trackers are running with the following command:: # ps aux | grep mogilefsd If you don't get a list of running processes the trackers are not running. Now that the MySQL database is setup and the tracker daemon is running we need to setup the storage server. Create a folder for the data:: mkdir /var/mogdata then create a config file ``/etc/mogilefs/mogstored.conf``:: httplisten=0.0.0.0:7500 mgmtlisten=0.0.0.0:7501 docroot=/var/mogdata Now you can start the storage server daemons:: # mogadm --trackers=localhost:6001 host add mogilestorage --ip=127.0.0.1 --port=7500 --status=alive You can confirm that your host(s) were added with the following command:: # mogadm --trackers=localhost:6001 host list mogilestorage [1]: alive IP: 127.0.0.1:7500 Now add a device:: mogadm --trackers=localhost:6001 device add mogilestorage 1 You can lost the devices like this:: mogile@vm1:/root$ mogadm --trackers=localhost:6001 device list mogilestorage [1]: alive used(G) free(G) total(G) dev1: alive 0.000 0.000 0.000 You'll then need to create a folder for that device:: mkdir /var/mogdata/dev1 Make sure it is owned by ``mogile``:: chown mogile:mogile /var/mogdata/dev1 Now you can start the storage daemon:: mogstored --daemon The following example would check all mogile components using the trackers at IP address 192.168.42.1 and 192.168.42.2, both listening on port 6001:: mogadm --trackers=localhost:6001 check You can specify multiple trackers to test by listing them as a comma separated list. Now it is all set up you can try it with some real data. Each file you store has to be in a domain and have a particular class so first we need to setup a domain and a class within that domain:: mogadm --trackers=localhost:6001 domain add testdomain Add a class to the domain:: mogadm --trackers=localhost:6001 class add testdomain testclass For the Python client:: apt-get install python python-pycurl Then download it:: wget http://www.albany.edu/~ja6447/mogilefs.py Create ``test.py``:: from mogilefs import * def test(): a = Admin(trackers=['localhost:6001']) print a.get_devices() good = open("/etc/motd").read() c=Client(domain='testdomain', trackers=['localhost:6001'], root='/var/mogdata') c.delete('/etc/motd') c.send_file('/etc/motd', '/etc/motd') assert(c.get_file_data('/etc/motd') == good) c.delete('/etc/motd_0') c.rename("/etc/motd", "/etc/motd_0") for x in range(10): c.delete('/etc/motd_%d' % (x+1)) c.rename("/etc/motd_%d" % x, "/etc/motd_%d" % (x+1)) data = c.get_file_data('/etc/motd_%d' % (x+1)) assert data == good if __name__ == '__main__': test() Then test it with:: python test.py You can also manipulate files via the command line with the ``mogadm`` tool. klipal discount retail buy zyban online cheap generic viagra levitra online female version of oxazepam whats a female xanax generic levitra cheapest lorazepam uk Fluoxetine purchase valium on line pfizer tramadol sales uk buy cialis canadian pharmacy oils for female diazepam female version of tenuate