Deploying an Echoprint server to EC2

For my current client work, I recently needed to deploy an instance of the open-sourced Echoprint server to the cloud. Echoprint is written in Python and uses Tokyo Cabinet and Tokyo Tyrant as it's data store. I found the only cloud hosting option available (given these technologies) was a custom Amazon EC2 environment. Now, I'm not an EC2 expert. And I must say their documentation and tutorials are both confusing.. and frequently out of date. It was a bit of a headache to get everything setup and configured.. so I thought I'd document it!

Here goes, if you'd like to setup the Echoprint open-sourced project on an Amazon EC2 instance... here's what you have to do. (As of Aug, 2013)

Setup an AWS account and launch an EC2 instance

Create (or login to) an Amazon AWS account and go to the management console. Navigate to your EC2 dashboard and click 'Launch Instance'. From here, choose the 'Quick Launch' wizard. This will be fine just to get us going. A few things to note:

1. Create a new public/private key pair for this server. We'll be using this later to setup ssh access. If you can download it now we'll get back to this later.

2. Server type. I chose the default Linux AMI, this is fine.

From here, finish up and click 'Continue' and then 'Launch'. This should launch an virtual server for you.. which is what we want!

From here, we need to be able to access this machine from our local computer. We'll need to setup SSH access for this. Download CLI tools. Confusingly these are called either the CLI tools or the API tools from different places in the AWS documentation. They are the same thing. Jeez!

I created a local directory where I would store all my EC2 stuff. I recommend doing the same. In a terminal execute:

> mkdir ~/.ec2

Unzip and move the bin and lib files from the CLI tools you just downloaded into this directory.
Move the key pair we created and downloaded during the wizard into here too.

You'll also need to update the permissions on that key-pair file.

> chmod 400 ~/.ec2/saskey-ec2.pem

Make sure the following environment variables are set. You'll have to get your AWS access and secret keys from the AWS console to set these variables. They are necessary. On a mac:

> vim ~/.bash_profile

Add:

export JAVA_HOME=$(/usr/libexec/java_home)
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
export AWS_ACCESS_KEY=AKIA***********FRS55A
export AWS_SECRET_KEY=u/ixxR*****************XsTfZl64I/H
export EC2_PRIVATE_KEY=$EC2_HOME/saskey-ec2.pem

Save and close (esc -> :wq!)

> source .bash_profile

Confirm everything is working by running the following command. Get the instance id from your ec2 instance console. It should look something like this: ec2-xx-xx-xx-xxx.compute-1.amazonaws.com.

Now we can ssh into our new linux server!

> ssh -i ~/.ec2/saskey-ec2.pem ec2-user@ec2-xx-xx-xx-xxx.compute-1.amazonaws.com

You should see something like this:

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

[ec2-user@ip-xx-xxx-xx-xx ~]$

Prepare This Server for Echoprint

We need to get the appropriate things onto the server:

Requirements for the server:

• java 1.6
• python 2.5 or higher
• simplejson (if python < 2.6)
• Tokyo Cabinet
• Tokyo Tyrant

Java should be on there. run > java -version

Python should be on there. run > python --version

I'm running Python 2.6.8, so no need to worry about the simplejson

We need to install Tokyo Cabinet - run the following:

> wget http://fallabs.com/tokyocabinet/tokyocabinet-1.4.48.tar.gz

> tar -xvf tokyocabinet-1.4.48.tar.gz

> cd tokyocabinet-1.4.48

> sudo yum install gcc

> sudo yum install zlib

> sudo yum install zlib-devel

> sudo yum install bzip2-devel.x86_64

> ./configure --enable-off64
> sudo make

> sudo make install

We need to install Tokyo Tyrant - run the following:

> wget http://fallabs.com/tokyotyrant/tokyotyrant-1.1.41.tar.gz

> tar -xvf tokyotyrant-1.1.41.tar.gz 

> cd tokyotyrant-1.1.41

> ./configure

> make

> sudo make install

Install the Echoprint project

Now we need the Echoprint project itself. I'm going to pull it from the Github project.

> sudo yum install git.x86_64

Go make a fork of the echoprint github project and git the github address. You'll need it below.

> cd /usr/local
> git clone https://github.com/your-git-id/echoprint-server

Now you've got the Echoprint project. Let's fire it up. Everything is installed and ready to startup.

Starting Solr (The Echoprint project uses Solr to index it's audiofingerprints)

> cd solr/solr
> java -Dsolr.solr.home=/usr/local/echoprint-server/solr/solr/solr/ -Djava.awt.headless=true -jar -DSTOP.KEY=YOURKEY -DSTOP.PORT=8079 start.jar

As a note, the logs will now output to: /usr/local/echoprint-server/solr/solr/logs

Stopping Solr (You may need this later)

> java -Dsolr.solr.home=/usr/local/echoprint-server/solr/solr/solr/ -Djava.awt.headless=true -jar -DSTOP.KEY=YOURKEY -DSTOP.PORT=8079 /usr/local/echoprint-server/solr/solr/start.jar --stop


Start Tokyo Tyrant

> sudo mkdir /var/ttserver
> sudo chown ec2-user /var/ttserver/
> cd /usr/local/sbin
> ls
> nohup ttservctl start &

Starting the server of Tokyo Tyrant
Executing: ttserver -port 1978 -dmn -pid /var/ttserver/pid

Done

Start Echoprint API Server

> sudo easy_install web.py

> cd /usr/local/echoprint-server/API

> nohup python api.py 8080 &

nohup because of ssh session

Make it Accessible to Outside world

If you'd like this accessible to the outside world.. to later do your importing/querying, you'll need to open up the port the Echoprint API server is running on. We have to tell our EC2 instance to do this. We do that by adding a rule to the security group's inbound rules, as shown below.

Now you're ready to import and query. Reference the Echoprint project documentation for instructions on how to do this!