If you see this error in your multithreaded boto3 python script:
ValueError: Unknown component: endpoint_resolver
then you might be using boto3 incorrectly for threading.
By default, boto3 uses the default session, which is not thread safe if you have more than one thread.
The fix is to ensure each thread starts its own session.
All explained quite well with a great example in the Boto3 Documentation:
In short, each thread should do it’s own:
session = boto3.session.Session()
s3 = session.resource('s3')
After searching around a bit to find a local (not online), simple and free ReStructuredText editor, I didn’t really find anything simple and easy that I liked, until I found the Atom editor. Why I like RST
better than Markdown is for another blog post.
With the launch of Atom 1.0
, it’s worth a look at what this new editor can do with its growing list of available plugins.
Lets go through how to rig this up.
- Install Atom
- Install Pandoc
- Install the atom-rst-preview plugin in Atom via Preferences / Install
- If on OSX, watch out for the Atom bug that will prevent the plugin from working (see the atom-rst-preview install instructions for the fix)
Copy the contents of an example RST file into Atom and hit ctrl-shift-r to see the live preview.
You can now edit and the preview will refresh when you type.
Nice and simple.
is probably the easiest go to tool for command line s3 transfers.
First install s3cmd using pip, install pip if it’s not already installed:
apt-get install python-pip
yum install python-pip
Next install the latest version available to pip, this is listed on the s3cmd PyPi page:
At the time of writing, the latest s3cmd version in PyPi is 1.5.0-alpha3, so install by using the exact version:
pip install s3cmd==1.5.0-alpha3
By default with pip install (at least on Ubuntu 14.04), you’ll only get version 1.0.1 which doesn’t handle IAM Roles. This is why you need to specify the latest version in the install.
Alternatively you could get the very latest version and install manually without pip if the pip version is lagging behind:
Now just create a ~/.s3cfg config with blank values to make s3cmd detect and use your IAM Role that you’ve assigned to your EC2 instance:
Now test it out:
s3cmd ls s3://bucket-you-should-have-access-to
Are you trying to install your new SSL certificate into AWS for use in an elastic load balancer but keep seeing this pesky error about PEM format:
Please ensure the private key is in PEM format
But you look at your private key and it looks like it’s PEM format already, because it starts with this text and it’s all ASCII readable:
-----BEGIN PRIVATE KEY-----
Well, your private key is not in X.509 PEM format just yet, because it should instead start with this line of text:
-----BEGIN RSA PRIVATE KEY-----
So, to convert it to X509 PEM format and stop all that wrong format guff, run this OpenSSL command (OpenSSL should be already installed on Linux or OSX):
openssl rsa -in yourwebsite_private.key -out pem-yourwebsite_private.key
where “yourwebsite_private.key” corresponds to your newly generated private ssl key and pem-yourwebsite_private.key is the new AWS pem formatted key that you will create.
Now it’s just a matter of uploading your new ssl files. If you’re savvy and are using the AWS CLI, you’ll use something like:
aws iam upload-server-certificate --server-certificate-name yourwebsite --certificate-body file://yourwebsite.crt --private-key file://pem-yourwebsite_private.key --certificate-chain file://yourwebsite_certificatechain.crt
For more information on using SSL certificates with Amazon AWS, see the official documentation:
Hope this helps people out 🙂
Ever wondered how to configure your autoscale groups to tag the instances they spin up?
I’m not sure this is supported from the AWS Web Console, but here’s how to do it from the command line…
UPDATE: Amazon have now implemented this feature from the web console: http://aws.amazon.com/blogs/aws/tag-your-auto-scaled-ec2-instances/
Using the AWS CLI (ensuring the CLI is configured correctly with your auth creds when you set it up,etc) you can simply set your autoscale groups to propagate tags to instances at launch time:
aws autoscaling create-or-update-tags --tags ResourceId="your-autoscale-group",ResourceType=auto-scaling-group,Key="Name",Value="name-for-all-your-instances",PropagateAtLaunch=True --region us-west-2
where “your-autoscale-group” is the name of the ASG you want to affect and “name-for-all-your-instances” is an example of setting the “Name” tag on newly initialised instances belonging to the ASG.
You can however propagate any tags you want to your instances using different Key names.