Tareque's Personal Portal

DES Alogrithm Illustrated

Disclaimer: All material provided in this post is for educational purpose only. Any other usage is strongly discouraged.

As part of a project for the Network Security course (Spring 2010) at GWU, I implemented DES or Data Encryption Standard in C. DES is an ancient encryption algorithm that is not used anymore since its effective key size is 56 bit which can be easily broken using brute force. However the same algorithm can be used to encrypt data in Triple-DES which has an effective key length of 168 bit and used everywhere.

According to Wikipedia:

The Data Encryption Standard (DES) is a block cipher (a form of shared secret encryption) that was selected by the National Bureau of Standards as an official Federal Information Processing Standard (FIPS) for the United States in 1976 and which has subsequently enjoyed widespread use internationally. It is based on a symmetric-key algorithm that uses a 56-bit key. The algorithm was initially controversial with classified design elements, a relatively short key length, and suspicions about a National Security Agency (NSA) backdoor. DES consequently came under intense academic scrutiny which motivated the modern understanding of block ciphers and their cryptanalysis.

You will be surprised by the lack of friendly and comprehendible DES implementations available as on the internet. Thankfully there are really great resources describing the algorithm step by step using examples. The article I used to write this implementation is The DES Algorithm Illustrated by J. Orlin Grabbe. It provides some interesting background on DES and NSA's attempt to create backdoors in the algorithm.

My implementation follows the article at each step. So if you are trying learn and implement DES, then you should be able to follow the code as you go through the article.

For example the section Step 1: Create 16 subkeys, each of which is 48-bits long maps to the function:

void generate_sub_keys(unsigned char* main_key, key_set* key_sets)

located here. Similarly the section Step 2: Encode each 64-bit block of data maps to the function

void process_message(unsigned char* message_piece, unsigned char* processed_piece, key_set* key_sets, int mode)

located here.

This also means that the code is not optimized in anyway. Key generation takes constant time. But encryption/ decryption O(N) where N is the file size. For example, encrypting the Firefox 3.6.3 Mac OSX installer sized at 18.6 MB takes about 30.66 seconds. Decryption takes slightly longer, 31.33 seconds. With a few quick optimizations I was able to cut this time in half. But the optimization severely sacrificed code readability. So I abandoned the plan of optimization. If you want speed, you will have to look for Assembly implementations. I found out that someone in MIT wrote a high performance version in assembly which is capable of en/decrypting files at ~20 MBps.

I have created a repository on GitHub where you can find all the source code, compiling and usage instructions. You can clone the repository or download the archived version des.zip.

Enabling pgAdmin Access to PostgreSQL on Cloud

Today I opened an account on Rackspace so that I can slowly migrate my sites from WebFaction. I have been using MySQL for my DB needs but this time decided to play with PostgreSQL. With v9.0 in the horizon with built in clustering support, Postgres might become the DB of choice on the cloud.

Anyway, I created my first server with CentOS 5.5 (hence instructions that will follow are guaranteed to work on CentOS 5.5 only, but will probably work for most other Linux distros) and installed PostgreSQL 8.4.3. Now I wanted to control this DB from pgAdmin III installed on laptop. How do I do that?

The solution was not apparent. There are few steps involved:

• Open up PostgreSQL port (default 5432) on the server
• Add rule in PostgreSQL's connection filter to allow request from remote machine
• Add server to pgAdmin and connect

Here I will describe each step in detail. All commands should be run as root or using sudo.

Open up PostgreSQL port

Incoming connections to port 5432 are rejected by default. So we have to open up this port using iptables configuration. This will not be making your system vulnerable to hackers. PostgreSQL has its own filters to control activities on its port.

We'll start by editing the configuration file:

emacs /etc/sysconfig/iptables

Then add (after all of the existing ACCEPT entries and before the final REJECT entry):

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT

Finally we'll restart iptables to load the new confuration.

/etc/init.d/iptables restart

Add rule in PostgreSQL's connection filter

PostgreSQL controls activities on its port (default: 5432) by using filter rules. We need to add a new rule to allow access to the DB from a remote IP.

First we need to edit the pg_hba.conf file where these rules exist. It's usually found within PostgreSQLs data directory. In my case:

emacs /opt/PostgreSQL/8.4/data/pg_hba.conf

The next step will work best if we have a static IP on the remote machine (In my case it's my laptop and it doesn't have a static IP. So I experimented with rules like 172.34.0.0/16 to allow more IPs, but it's not recommended) Let's say the current IP is 172.34.228.5. We'll add the following line at the end of the rules:

# Connection from my laptop
host    all         all         172.34.228.5/32         md5

Save and exit. Next we'll restart PostgreSQL to take new filter into account.

sudo -u postgres ./pg_ctl restart -D /opt/PostgreSQL/8.4/data

Now we can add the cloud server to pgAdmin with port and other information specified. You should have specified a password for your postgres account when installing postgres. This password will provide the final layer of security. PostgreSQL will not accept connection to the DB if you all of the above but provide incorrect password.

Django Deployment Strategy

Presentation on Xen and the Art of Virtualization

Xen is one of the most popular & widely used virtual machine monitor. It was introduced in the 2003 publication Xen and the Art of Virtualization by Paul Barham et al. of University of Cambridge Computer Lab. Here's part of the abstract for a quick overview:

This paper presents Xen, an x86 virtual machine monitor which allows multiple commodity operating systems to share conventional hardware in a safe and resource managed fashion, but without sacrificing either performance or functionality. This is achieved by providing an idealized virtual machine abstraction to which operating systems such as Linux, BSD and Windows XP, can be ported with minimal effort.

Xen defines a primary virtual machine called the "Domain 0" which has direct access to hadware and also manages the other virtual machines running on the system. The Domain 0 is also called the Xen Hypervisor. Running multiple virtual machines on a single physical machine not only allows us to overcome the lack of environment/ user separation facility in traditional OSes but also ensures maximum utilization of available hardware. The latter has given rise to the cloud computing culture where many users share the same hardware, not necessarily simultaneously and enjoy the benefits of low cost hosting.

Unlike most academic projects, Xen has become a commercial success and now being used by different cloud computing service providers e.g. Amazon EC2. Xen is an open source project and have been enjoying contributions from hardware vendors like Intel and AMD, who have made it possible for Xen to run different OSes in virtual environments without any modification and with very little overhead.

As an assignment for Dr. Gabriel Parmer's Spring 2010 Advanced Operating System course, I reviewed the aforementioned paper on Xen and presented my findings at the class.

RabbitMQ & Nodedown Errors

If you are experiencing issues when starting RabbitMQ server or using RabbitMQ Control Script - rabbitmqctl and the error is:

Error: unable to connect to node 'rabbit@mydomain': nodedown

Then open up /etc/host file and add the following line:

127.0.0.1    mydomain

Also if you have started RabbitMQ as a service (being root) and you either see the nodedown error or the following error:

"Error when reading /some_path/.erlang.cookie: eacces"

then copy the default erlang cookie from /var/lib/rabbitmq:

cp -f /var/lib/rabbitmq/.erlang.cookie /some_path/.erlang.cookie

Running Your Own MySQL Instance on WebFaction

One of my recent projects required me to write Triggers. Only with version 5.1.6 MySQL implemented Trigger/ Routine privileges. In all prior versions that support triggers, require you to have root privileges to create triggers/ routines. WebFaction provides version 5.0.77 by default. So I needed to install and run the latest version of MySQL on my WebFaction account. You might have other reasons to run your own MySQL instance, for example, greater control over your database.

likebike on WebFaction forum has an excellent starting point for having your own MySQL server running. Click here to read his post.

But it is outdated and requires you to compile MySQL source and by doing that you will end up with a limited set of database engines. Triggers, routines and foreign key constraints require the InnoDB engine which does not get installed if you compile and install from MySQL source.

Here's the step by step guide:

Reserve a port number by going into the WebFaction control panel and creating a new app of type "Custom App listenting on port". Note down the port number.

Go to your home directory:

cd ~/

Download MySQL binary for unix/ linux from the official site. Get Linux (x86) version.

wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.41-linux-i686-glibc23.tar.gz/from/http://mysql.he.net/

Extract the file.

tar xvf mysql-5.1.41-linux-i686-glibc23.tar.gz

Rename the extracted folder to mysql (assuming you don't have one created already):

mv mysql-5.1.41-linux-i686-glibc23 mysql

Create a file name my.cnf in the mysql directory:

touch ~/mysql/my.cnf

Edit the file and put the following information in the file:

[mysqld]
port=$MY_PORT
socket=$HOME/mysql/mysql.sock
basedir=$HOME/mysql
datadir=$HOME/mysql/data

[client]
port=$MY_PORT
socket=$HOME/mysql/mysql.sock

Install the default schema:

cd ~/mysql/scripts
./mysql_install_db --basedir=$HOME/mysql --datadir=$HOME/mysql/data

Go to the mysql binary directory:

cd ~/mysql/bin

Start the MySQL server (parameter order matters):

./mysqld_safe --defaults-file=$HOME/mysql/my.cnf --ledir=$HOME/mysql/bin &

Leave it running in the background by pressing ctrl+c. You should be able to see your MySQL process by typing:

ps -u username -O pid,rss,command | grep 'mysql'

Change your root password:

./mysqladmin -u --socket=$HOME/mysql/mysql.sock root password 'newpassword'

Now to get your django application to recognize and connect to this database, you have to change 'DATABASE_HOST' setting in your django settings file. By default you put 'localhost' here. But the have django recognize your alternate database instance, you can use the socket path instead. So add/ replace the following line in your settings.py:

DATABASE_HOST = '/home/username/mysql/mysql.sock'

If you need to access the MySQL prompt, use the following command:

~/mysql/bin/mysql --socket=$HOME/mysql/mysql.sock -u root -p

You can also create an alias in your .bashrc profile using the command above for easy access to your MySQL prompt.

Enjoy. Remember running MySQL instance will take up 15MB ~ 20MB of memory. So it might not be an option if you're running too many Apache threads/ django instances.

Also WebFaction might kill your MySQL daemon if you're above the memory limit. In order to prevent that, you can setup a cron job to try to start your MySQL instance every 10/ 15 minutes.

Memory Usage on WebFaction

If you're hosting your website on WebFaction, it's very easy to spin up multiple django instances and completely use up allocated memory, specially if you have the 80MB limit.

WebFaction panel does not let you keep track of Memory usage, just disk and bandwidth. Here's a little shell command, that will calculate your memory usage in KiloBytes:

ps -u username -O pid,rss,command | awk '{sum += $3} END {print sum}'

Remove the pipe and remainder of the command to see a detailed listing of Process IDs, Memory Usage and the Commands that invoked the applications using memory.

If you have too many httpd processes running, you may have set you ServerLimit too high. If you're on the 80MB limit host plan and have 2/3 django instances running, set the following in your Apache httpd.conf:

ServerLimit 3

Introduction to django-config

This year at DjangoCon, I did a lightning talk presentation on django settings management. django-config allows you to maintain user/ server specific configurations for django that can be switched on the fly. The presentation received a spotlight on SlideShare

The code can be checked out from github

Processing Django Form Errors

    dict([(field, [unicode(error) for error in errors]) for field, errors in form.errors.items()])