Streaming File Upload with Erlang and Mochiweb Multipart Post
Here’s one way to set up a simple Mochiweb server to handle a file upload using Erlang.
On Debian you’ll need to install the following:
sudo apt-get install build-essential libncurses-dev libssl-dev m4 subversion
(I’ve also got autoconf, automake and libgd-dev so you might need those too).
First install the latest version of Erlang:
wget http://erlang.org/download/otp_src_R12B-0.tar.gz tar zxfv otp_src_R12B-0.tar.gz cd otp_src_R12B-0
I already have Erlang installed as part of the Debian distribution so I install this version to my home directory so it doesn’t conflict:
./configure --prefix=/home/james
You’ll get an error similar to this but we won’t be using these features so we can egnore the warning:
********************************************************************* ********************** APPLICATIONS DISABLED ********************** ********************************************************************* jinterface : No Java compiler found odbc : No odbc library found percept : libgd not working *********************************************************************
Now make and install Erlang:
make make install
This takes about 3 minutes to complete on a fairly modern dual core server.
You’ll probably want the html and man pages too:
cd /home/james/lib/erlang wget http://erlang.org/download/otp_doc_man_R12B-0.tar.gz wget http://erlang.org/download/otp_doc_html_R12B-0.tar.gz gunzip -c otp_doc_man_R12B-0.tar.gz | tar xf - gunzip -c otp_doc_html_R12B-0.tar.gz | tar xf - rm otp_doc_man_R12B-0.tar.gz otp_doc_html_R12B-0.tar.gz
Now that Erlang is set up get the latest trunk of Mochiweb:
svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb
You’ll need to compile it:
export PATH=/home/james/bin:/home/james/lib:$PATH cd mochiweb make
Now setup a project called file_upload like this:
cd scripts chmod a+x new_mochiweb.erl ./new_mochiweb.erl file_upload /home/james/
Compile it and run it:
make ./start-dev.sh
The server will be accessible on port 8080. You can stop it by pressing Ctrl+c and then typing the letter a followed by return. Try visiting http://yourserver.com:8000/index.html and you should see the message MochiWeb running..
You now need to modify the src/file_upload_web.erl file for our file upload application. Underneath the -export() line near the top add this:
-record(state, {filename, file}).
callback(Next, State) ->
case Next of
{headers, Headers} ->
% Find out if it is a file
[ContentDisposition|_] = Headers,
NewState = case ContentDisposition of
{"content-disposition", {"form-data",[{"name",_},
{"filename",Filename}]}} ->
#state{filename="/tmp/"++Filename};
_ ->
State
end,
fun(N) -> callback(N, NewState) end;
{body, Data} ->
if State#state.filename =/= undefined ->
if State#state.file =/= undefined ->
file:write(State#state.file, Data),
NewState = State;
true ->
case file:open(State#state.filename, [raw,write]) of
{ok, File} ->
file:write(File, Data),
NewState = State#state{file=File};
{error, Error} ->
io:format(
"Couldn't open ~p for writing, error: ~p~n",
[State#state.filename, Error]),
NewState=State,
exit(could_not_open_file_for_writing)
end
end;
true ->
NewState = State
end,
fun(N) -> callback(N, NewState) end;
body_end ->
if State#state.file =/= undefined ->
file:close(State#state.file);
true ->
ok
end,
fun(N) -> callback(N, #state{}) end;
_ ->
fun(N) -> callback(N, State) end
end.
Then replace the loop function with this:
loop(Req, _) ->
case Req:get(method) of
Method when Method =:= 'GET'; Method =:= 'HEAD' ->
Req:ok({"text/html", [], <<"<html><body><h1>File Upload</h1>
<form enctype=\"multipart/form-data\" action=\"/\" method=\"post\">
<label for=\"file\">File:</label>
<input type=\"file\" name=\"file\" id=\"file\"/>
<input type=\"submit\" name=\"upload\" value=\"Upload\" />
</form>
</body></html>">>});
'POST' ->
Callback = fun(N) -> callback(N, #state{}) end,
mochiweb_multipart:parse_multipart_request(Req, Callback),
Req:ok({"text/html", [], <<"<html><body><h1>File Upload</h1>
<p>Uploaded successfully.</p>
</body></html>">>});
_ ->
Req:respond({501, [], []})
end.
You’ll need to build the new code by running make again in the file_upload directory. The server should automatically restart with the message Reloading file_upload_web … ok. - if it doesn’t you’ll need to manually stop and start the server.
If you run the application again you should now have a simple file upload application which saves files to /tmp.
Nginx Proxying to Pylons with SSL on Debian Etch
The easy way to install Nginx is like this:
sudo aptitude install nginx
The problem is that this installs an old version (0.4) and doesn’t have SSL support built in. Nginx is designed to be super fast so and modules need to be compiled in manually. This means you need remove the Nginx package if you want SSL support.
If you try sudo apt-get remove –purge nginx you get this error:
Reading package lists... Done Building dependency tree... Done The following packages will be REMOVED nginx* 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. Need to get 0B of archives. After unpacking 524kB disk space will be freed. Do you want to continue [Y/n]? y (Reading database ... 26362 files and directories currently installed.) Removing nginx ... Stopping nginx: invoke-rc.d: initscript nginx, action "stop" failed. dpkg: error processing nginx (--purge): subprocess pre-removal script returned error exit status 1 Starting nginx: 2007/12/07 14:45:03 [emerg] 29441#0: unknown directive "ssl" in /etc/nginx/nginx.conf:30 invoke-rc.d: initscript nginx, action "start" failed. dpkg: error while cleaning up: subprocess post-installation script returned error exit status 1 Errors were encountered while processing: nginx E: Sub-process /usr/bin/dpkg returned an error code (1)
The solution is to edit /var/lib/dpkg/info/nginx.prerm and comment out the line with invoke-rc.d which gives the stop statement for nginx. If you run the command again it will remove successfully, purging your nginx.conf config file too. If this isn’t what you want make a backup first.
You can then remove your logs with:
sudo rm -rf /var/log/nginx/
Now to install Nginx 0.6 from source. Fisrt you will need some build tools:
sudo aptitude install build-essential
Then we need some libraries:
sudo aptitude install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev
Check what the latest version is at http://nginx.net. At the time of writing the latest stable version is 0.5.33 so download that:
wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz tar zxfv nginx-0.5.33.tar.gz cd nginx-0.5.33
Now you are ready to compile Nginx. As mentioned earlier many of the options are set at compile time so have a look at the Compile Time Options page and decide which you need. We are going to set two options. The first is:
--with-http_ssl_module
This enables the SSL module. The second option is to customise where Nginx is installed to. The default is /usr/local/nginx but this means the Nginx binary will be /usr/local/nginx/sbin/nginx which isn’t on the PATH. A better place is /usr/local/sbin where it is easily accessible by root or by users with sudo access:
--sbin-path=/usr/local/sbin
Now we are ready to configure:
./configure --sbin-path=/usr/local/sbin --with-http_ssl_module
There is a useful configuration summary:
nginx path prefix: "/usr/local/nginx" nginx binary file: "/usr/local/sbin" nginx configuration file: "/usr/local/nginx/conf/nginx.conf" nginx pid file: "/usr/local/nginx/logs/nginx.pid" nginx error log file: "/usr/local/nginx/logs/error.log" nginx http access log file: "/usr/local/nginx/logs/access.log" nginx http client request body temporary files: "/usr/local/nginx/client_body_temp" nginx http proxy temporary files: "/usr/local/nginx/proxy_temp" nginx http fastcgi temporary files: "/usr/local/nginx/fastcgi_temp"
Then compile and install:
make sudo make install
The first thing I do next is create a backup of the original file:
sudo cp /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf.bak
Then start the server:
sudo /usr/local/sbin/nginx
If you visit http://yourdomain.com you should see the Nginx welcome message Welcome to nginx!. If you don’t it might be because you have another server such as Apache running.
To kill the server you can use the pid file:
sudo kill `cat /usr/local/nginx/logs/nginx.pid`
Notice this uses the ` character (normally underneath Esc) not the ‘ character. Obviously it is nicer to have an init.d setup so you can start and stop Nginx the same way as Apache and other servers. We’ll use the script from Slicehost which is itself based on the Debian package:
wget http://articles.slicehost.com/assets/2007/10/19/nginx sudo chmod +x nginx sudo mv nginx /etc/init.d
Now we can add this script to the runlevels:
sudo /usr/sbin/update-rc.d -f nginx defaults
Now you can start, stop and restart Nginx with these commands as normal:
sudo /etc/init.d/nginx start sudo /etc/init.d/nginx stop sudo /etc/init.d/nginx restart
Now let’s setup Nginx to proxy to a Pylons application. Replace these lines in /usr/local/nginx/conf/nginx.conf:
location / {
root html;
index index.html index.htm;
}
with these:
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_pass http://127.0.0.1:5000;
proxy_redirect default;
}
Update: Added X-Forwarded-Port and X-Forwarded-Host to the above since I use them both so much.
This will proxy all requests to another server running on port 5000, for example a running Pylons application. To test this you should start your Pylons application (making sure debug is set to false if this is a production setup):
paster serve development.ini
Now restart Nginx:
sudo /etc/init.d/nginx restart
and f you visit the site you should see your Pylons application.
Now for the SSL:
sudo aptitude install openssh-server
First create a the key, pem and certificate files:
openssl genrsa 1024 > host.key chmod 400 host.key openssl req -new -x509 -nodes -sha1 -days 365 -key host.key > host.cert cat host.cert host.key > host.pem chmod 400 host.pem
Make sure the Common Name is the same as the domain name the certificate is for.
Now we need to setup SSL on Nginx. This is described in detail here but it really just requires making the server section look like this:
worker_processes 1;
http {
...
server {
listen 443;
ssl on;
ssl_certificate /path/to/host.pem;
ssl_certificate_key /path/to/host.key;
keepalive_timeout 70;
}
}
Restart Nginx again:
sudo /etc/init.d/nginx restart
Now if you visit http://yourdomain.com you’ll get no response but if you visit https://yourdomain.com you’ll get a certificate warning because the SSL certificate is not signed by a certificate authority. Click OK and you will have access to your application.
The final part of the setup is to setup a static file which redirects from the http version of the site to the https version. The easy way to do this is by setting up another server listening on port 80 and creating an error 404 page which redirects to the https version. Add this to the nginx config just before the existing server:
server {
listen 80;
server_name yourdomain.com;
error_page 404 /404.html;
location = /404.html {
root /path/to/directory/containing/404/doc/;
}
}
Then create a 404.html file which looks something like this but adjusted for your URL instead of yourdomain.com:
<html> <head> <meta http-equiv="refresh" content="2;URL=https://yourdomain.com/"> </head> <body> <h2>Redirecting</h2> <p>You are being redirected to the <a href="https://yourdomain.com/">secure version of this site.</a></p> </body> </html>
Then restart Nginx again.
You should now have a secure Pylons app on the https port and a redirect page on the http port.
Further Reading:
http://articles.slicehost.com/2007/10/19/debian-etch-installing-nginx http://ubuntuforums.org/showthread.php?t=453053 http://sudhanshuraheja.com/2007/09/remove-nginx-from-ubuntu-fiesty-fawn.html http://www.rkblog.rk.edu.pl/w/p/pylons-and-nginx/
Subversion over SVN+SSH on Debian
The traditional way of setting up subversion is via Apache but what if you don’t want to install Apache? In that case you can use svnserve but what if your host won’t let you run a persistant server? In that case you can create a linux system account for every user and give them induvidual SSH logins. You then give your users a URL starting svn+ssh:// instead of the usual http://. The Subversion client recognises this form of URL and invokes a local ssh process, connecting to the host, authenticating as the user, then spawning a private svnserve process on the remote machine running as that user. The svnserve command is being invoked in tunnel mode (-t) and its network protocol is being tunneled over the encrypted connection by ssh, the tunnel-agent. svnserve is aware that it’s running as the user, and if the client performs a commit, the authenticated username will be attributed as the author of the new revision.
This is all well and good but you might not wa nt to have to create user accounts for each of the users you want to give access to subversion. In this case you can create one system user account and use a different SSH private key for each user you want to grant access to. You then setup the ~/.ssh/authorized_keys file so that each user only has access to svnserve and that each svnserve process is started with their particular username so that all the commits are handled correctly. This is only possible with subversion 1.1.0 and above.
This is what we are going to do for two users, james and mike. james is a linux user, mike is a Windows user.
First let’s setup the server:
sudo apt-get install subversion openssh-server
Then create a subversion user svn, a home directory, an .ssh directory and finally set the correct permissions:
sudo useradd svn sudo mkdir /home/svn sudo mkdir /home/svn/.ssh sudo chown -R svn:svn /home/svn
If you intend to be able to login as svn for testing purposes later you should set a password:
sudo passwd svn
Create the directory for the repositories:
sudo mkdir /var/svn
Setup a repo repository and commit the first import:
sudo mkdir /var/svn/repo sudo mkdir /tmp/repo sudo mkdir /tmp/repo/branches sudo mkdir /tmp/repo/tags sudo mkdir /tmp/repo/trunk sudo svnadmin create /var/svn/repo sudo svn import /tmp/repo file:///var/svn/repo -m "initial import" sudo rm -rf /tmp/repo
Change the permissions so that the svn user and svn group have access:
sudo chown -R svn:svn /var/svn/repo
Set the permissions to 770 and the umask to 2 so that any files are created have the correct permissions:
sudo chmod 2770 -R /var/svn/repo
At this point you can test the setup manually. As a user other than svn try this:
svn co svn+ssh://svn@localhost/var/svn/repo .
This should prompt you for the password a couple of times and then checkout the repository.
We are half way there, next we need to set up two SSH public-private key pairs, one for Mike, one for James. As the svn user run:
ssh-keygen -t rsa -b 1024 -f mike.key ssh-keygen -t rsa -b 1024 -f james.key
With each command you can just press <Enter> twice unless you want to set a password. You will now have the files mike.key, mike.key.pub, james.key and james.key.pub in the current directory. The .key files are the private keys. They should be given securely to Mike and James repsectively. Anyone who gets hold of the files will have the same permissions to the repository as James and Mike so they should be treated with the same respect as a password and not posted online!
Since Mike is on Windows he will need WinSCP, PuTTY, PuTTYgen and TortoiseSVN (the latter requires adminsitrator access to install).
Here’s what he does:
Use WinSCP to copy the mike.key file to his desktop if he didn’t recieve it any other way
Open PuTTYgen and from the Conversions menu, select Import, then browse to and select mike.key. Leave the passphrase boxes blank and select Save private key. Click Yes on the confirmation box, and save the key with a .ppk extension, like mike.ppk. This is now a key that PuTTY can use.
Start PuTTY again. Fill out the hostname box, then scroll down to SSH > Auth. Browse for the mike.ppk file. Then go back up to Sesson, give the session a name (for example mike) and save it. Test by logging in with the username ssh. You should see something like this:
login as: svn Authenticating with public key "imported-openssh-key" ( success ( 1 2 ( ANONYMOUS EXTERNAL ) ( edit-pipeline svndiff1 absent-entries ) ) )
With this in place you can set up TortoiseSVN with the key, [I haven’t tested this next bit yet] First, create a new folder, then right-click it and select SVN Checkout. In the dialog box, enter this in the URL field svn+ssh://svn@mike/var/svn/repo. svn+ssh is how TortoiseSVN will access your server, svn@mike is your credentials for logging in (mike is the session we set up using the keys in PuTTY), and /var/svn/repo is where the repository is located on the server. Later on in this tutorial we’ll configure the server so this can be accessed as svn+ssh://svn@mike/repo.
From James’s point of view the process is a bit simpler becasue on Linux the .key file doesn’t need to be converted to a .ppk file:
Create a ~/.ssh directory:
mkdir ~/.ssh
Copy the private key to ~/.ssh/id_dsa you don’t already have it:
scp svn@example.com:/home/svn/james.key ~/.ssh/id_dsa
Now that the private keys are set up it is time to setup the public keys on the subversion server. Copy the contents of mike.key.pub and james.key.pub to /home/svn/.ssh/authorized_keys so that the each publick key is on a separate line:
touch .ssh/authorized_keys cat mike.key.pub >> .ssh/authorized_keys cat james.key.pub >> .ssh/authorized_keys
It should look something like this but with lots of characters in place of …:
ssh-rsa AAA...acFHU= svn@example ssh-rsa AAA...acBDH= svn@example
The three columns are TYPE KEY COMMENT. Since the last part is a comment it makes sense to change it to the names of the users so you remember which line referrs to who:
ssh-rsa AAA...acFHU= mike ssh-rsa AAA...acBDH= james
This is all well and good but at the moment Mike and James can also SSH directly into the svn account so we want to restrict their access. Each line in the authorized_keys file can also have a command="COMMAND" section at the start. The subversion client executes svnserve -t when it SSHs in but you can also specify the executable to use explicitly:
command="/usr/bin/svnserve -t" ssh-rsa AAA...acFHU= mike command="/usr/bin/svnserve -t" ssh-rsa AAA...acBDH= james
Now you might want to limit the users to a particular repository, you can do this with the -r option:
command="/usr/bin/svnserve -t -r /var/svn/" ssh-rsa AAA...acFHU= mike command="/usr/bin/svnserve -t -r /var/svn/" ssh-rsa AAA...acBDH= james
James and Mike can access the repo repository with svn+ssh://svn@localhost/repo instead of svn+ssh://svn@localhost/var/svn/repo
You will probably want to limit the other things the users can do with the svn account so you’d probably add no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding to the command like this:
command="/usr/bin/svnserve -t -r /var/svn/",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acFHU= mike command="/usr/bin/svnserve -t -r /var/svn/",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acBDH= james
So far, the svnserve command still doesn’t know which username it is supposed to be using so we add the –tunnel-user option to tell it explicitly:
command="/usr/bin/svnserve -t -r /var/svn/ --tunnel-user=mike",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acFHU= mike command="/usr/bin/svnserve -t -r /var/svn/ --tunnel-user=james",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acBDH= james
At this point everything should now work as expected. Both James and Mike should be able to access and commit to the repository without needing a password and their commits should be logged as james and mike respectively because they are using different private keys and we are using the –tunnel-user approach. No other users should be able to access the repository except the svn user if you know the password.
Further reading:
http://svnbook.red-bean.com/en/1.1/ch06s03.html http://allyourtech.com/content/articles/23_12_2005_setting_up_subversion_and_tortoisesvn.php http://allyourtech.com/content/articles/24_12_2005_ditching_the_password_prompts_in_tortoisesvn.php http://svn.collab.net/repos/svn/trunk/notes/ssh-tricks http://svn.haxx.se/dev/archive-2004-03/0253.shtml http://bitworking.org/news/Getting_subversion_svn_ssh____to_work_with_PuTTY http://www.linuxfromscratch.org/blfs/view/svn/server/svnserver.html
For testing it is also possible to set the $SVN_SSH variable specifying the private key to use with -i so that you can try connecting as different users. For example:
export SVN_SSH="ssh -i /home/james/.ssh/mike.key