Multithreading boto3

If you see this error in your multithreaded boto3 python script:

ValueError: Unknown component: endpoint_resolver

then you might be using boto3 incorrectly for threading.

By default, boto3 uses the default session, which is not thread safe if you have more than one thread.

The fix is to ensure each thread starts its own session.

All explained quite well with a great example in the Boto3 Documentation:

In short, each thread should do it’s own:

session = boto3.session.Session()
s3 = session.resource('s3')

Using Atom as an RST editor with live preview

After searching around a bit to find a local (not online), simple and free ReStructuredText editor, I didn’t really find anything simple and easy that I liked, until I found the Atom editor. Why I like RST better than Markdown is for another blog post.
 atom editor logo
With the launch of Atom 1.0, it’s worth a look at what this new editor can do with its growing list of available plugins.
One such plugin for Atom is atom-rst-preview by Loren Segal, which lets you live preview your ReStructuredText as you edit.
Lets go through how to rig this up.
The atom-rst-preview Readme file on github is fairly self explanatory, but in short:

  • Install Atom
  • Install Pandoc
  • Install the atom-rst-preview plugin in Atom via Preferences / Install
  • If on OSX, watch out for the Atom bug that will prevent the plugin from working (see the atom-rst-preview install instructions for the fix)

Copy the contents of an example RST file into Atom and hit ctrl-shift-r to see the live preview.

atom rst preview example

You can now edit and the preview will refresh when you type.

Nice and simple.

How to use s3cmd with AWS IAM Roles

S3Cmd is probably the easiest go to tool for command line s3 transfers.
This is how you use it with AWS IAM Roles:
First install s3cmd using pip, install pip if it’s not already installed:
apt-get install python-pip
yum install python-pip
Next install the latest version available to pip, this is listed on the s3cmd PyPi page:
At the time of writing, the latest s3cmd version in PyPi is 1.5.0-alpha3, so install by using the exact version:
pip install s3cmd==1.5.0-alpha3
By default with pip install (at least on Ubuntu 14.04), you’ll only get version 1.0.1 which doesn’t handle IAM Roles. This is why you need to specify the latest version in the install.
Alternatively you could get the very latest version and install manually without pip if the pip version is lagging behind:
Now just create a ~/.s3cfg config with blank values to make s3cmd detect and use your IAM Role that you’ve assigned to your EC2 instance:
access_key =
secret_key = 
security_token =
Now test it out:
s3cmd ls s3://bucket-you-should-have-access-to

Converting SSL private key to x509 PEM format for Amazon AWS

Are you trying to install your new SSL certificate into AWS for use in an elastic load balancer but keep seeing this pesky error about PEM format:

Please ensure the private key is in PEM format

But you look at your private key and it looks like it’s PEM format already, because it starts with this text and it’s all ASCII readable:


Well, your private key is not in X.509 PEM format just yet, because it should instead start with this line of text:


So, to convert it to X509 PEM format and stop all that wrong format guff, run this OpenSSL command (OpenSSL should be already installed on Linux or OSX):

openssl rsa -in yourwebsite_private.key -out pem-yourwebsite_private.key

where “yourwebsite_private.key” corresponds to your newly generated private ssl key and pem-yourwebsite_private.key is the new AWS pem formatted key that you will create.

Now it’s just a matter of uploading your new ssl files. If you’re savvy and are using the AWS CLI, you’ll use something like:

aws iam upload-server-certificate --server-certificate-name yourwebsite --certificate-body file://yourwebsite.crt --private-key file://pem-yourwebsite_private.key --certificate-chain file://yourwebsite_certificatechain.crt

For more information on using SSL certificates with Amazon AWS, see the official documentation:

Hope this helps people out 🙂


Tagging with autoscaling groups

Ever wondered how to configure your autoscale groups to tag the instances they spin up?

I’m not sure this is supported from the AWS Web Console, but here’s how to do it from the command line…

UPDATE: Amazon have now implemented this feature from the web console:

Using the AWS CLI (ensuring the CLI is configured correctly with your auth creds when you set it up,etc) you can simply set your autoscale groups to propagate tags to instances at launch time:

aws autoscaling create-or-update-tags --tags ResourceId="your-autoscale-group",ResourceType=auto-scaling-group,Key="Name",Value="name-for-all-your-instances",PropagateAtLaunch=True --region us-west-2

where “your-autoscale-group” is the name of the ASG you want to affect and “name-for-all-your-instances” is an example of setting the “Name” tag on newly initialised instances belonging to the ASG.

You can however propagate any tags you want to your instances using different Key names.

Happy clouding!