Posts in Category: uncategorized

S3 – Use IAM role to grant S3 access to ec2 instances

Not sure if this is article works (at least it works without setting up any bucket policies). Need to investigate further.

I recently discovered that you don’t need to set up S3 bucket policies in order to give an EC2 access to an s3 bucket (or folder). Say you want to copy files to an s3 bucket from an EC2 instance:

$ aws s3 cp /path/to/testfile.txt s3://s3-bucketname/path/to/s3-bucket-folder

Then you need to do the following:

1. attach an IAM role to your ec2 instance
2. create the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
															

The screen command

useful for:

collaboration:

1. share screens with colleages
2. long running jobs. e.g. cp 100GB to an NFS. You can monitor progress by going back the screen.

https://www.lynda.com/Linux-tutorials/Manage-terminal-sessions-screen/618702/729627-4.html


Register external service to consul

https://www.consul.io/docs/guides/external.html

https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service

$ curl -X PUT -d '{
"Node": "aws-ntp",
"Address": "169.254.169.123",
"Service": {
"Service": "ntp"
}
}' http://localhost:8500/v1/catalog/register

nslookup of ntp.service.{{domain}} should now work.

you can then populate /etc/ntp.conf with ‘ntp.service.{{domain}}’ using consul templates.

In case you want to deregister this service from consul, run:

$ curl --request PUT --data @deregister.json http://localhost:8500/v1/catalog/deregister

where:

$ cat deregister.json
{
"Node": "aws-ntp",
"Address": "169.254.169.123",
"Service": {
"Service": "ntp"
}
}


Editing the httpd.conf file using Augeas

Sometimes you might want to edit the httpd.conf file using a shell script. For example let’s say we have the following file on our CentOS 7 machine:

$ cat /etc/httpd/conf/httpd.conf
ServerRoot "/etc/httpd"
Listen 80
Include conf.modules.d/*.conf
User apache
Group apache
ServerAdmin root@localhost

    AllowOverride none
    Require all denied

DocumentRoot "/var/www/html"

    AllowOverride None
    Require all granted


    Options Indexes FollowSymLinks
    AllowOverride None                    # I WANT TO EDIT THIS LINE ONLY
    Require all granted


    DirectoryIndex index.html


    Require all denied

ErrorLog "logs/error_log"
LogLevel warn

    LogFormat "%h %l %u %t \"%r\"															


puppet performance tuning

The latest version of PE 2016.4 has the capability to monitor the heap memory as a feature of puppet server 2.6.

https://docs.puppet.com/puppetserver/2.6/status-api/v1/services.html#example-request-and-response-for-a-debug-level-get-request

this guide:

https://puppet.com/blog/puppet-server-advanced-memory-debugging

https://docs.puppet.com/pe/latest/install_multimaster.html

https://support.puppet.com/hc/en-us/articles/225049688

https://docs.puppet.com/puppetserver/latest/tuning_guide.html#number-of-jrubies

Thundering herd test:

After you’ve added hundreds of nodes to your deployment you may notice that your agents are running slow or timing out. When hundreds of nodes check in simultaneously to request a catalog, it might cause a so-called thundering herd of processes that causes CPU and memory performance to suffer. To verify that you have a thundering herd condition, you can run a query on the PuppetDB node (the master in a monolithic installation) to show how many nodes check in per minute.

Log into the PuppetDB node(the master in a monolithic installation) as the pe-postgres user.

Open the PostgreSQL command line interface by running sudo su – pe-postgres -s


Puppet – Using AWS web console as Puppet’s external node classifier (ENC)

This is a script I wrote that queries the ec2 tags of an aws console, in order to figure out what environment a node belongs to, and what class to assign to it.

#!/bin/bash

# https://docs.puppetlabs.com/guides/external_nodes.html
# http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

export AWS_ACCESS_KEY_ID=xxxxxxxxxxxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxx
export AWS_DEFAULT_REGION=xxxxxxx

instanceid=`echo $1 | awk -F"_" '{print $NF}'` # $1 will match the certname

env=`/bin/aws --output text ec2 describe-instances --instance-ids $instanceid | grep '^TAGS' | grep 'env' |   awk '{print $NF}'`
role=`/bin/aws --output text ec2 describe-instances --instance-ids $instanceid | grep '^TAGS' | grep 'role' |   awk '{print $NF}'`

#aws ec2 describe-instances --instance-ids $instanceid > /tmp/enc-log.txt
#echo "puppet run occured at `date`" > /tmp/enc-log.txt
#echo "The first param value is: $1" >> /tmp/enc-log.txt
#echo "The environment is: $env" >> /tmp/enc-log.txt
#echo "The puppet role is: $role" >> /tmp/enc-log.txt

echo '---'
echo 'classes:'
echo "   - roles::$role"
echo "environment: $env"   
echo "parameters:"
echo															

AWS – SSL termination on the ELB

http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/using-elb-listenerconfig-quickref.html

 

https://blog.qruizelabs.com/2014/06/06/ssl-aws-elb/

In my case I replaced:

!/elbcheck.html

with:

^(.*)$