VPC creation, launching instance inside VPC using EC2 portal and Bootstrap EC2 instance using VPC opitons

• Launching instance inside VPC using portal

Follow following link to

http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-create-vpc.html

1) Create VPC

2) Create security group for VPC

3) Launch instance inside VPC

4) Assigning an Elastic IP Address to your instance

• Creating vm and bootstraping node which is created under VPC follow following steps

1) Create VPC as per instruction given in Step I of http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-create-vpc.html

2) Crete Security group for VPC in Step 2 i.e. http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-create-security-group.html

Note: Make sure you are creating security group inside VPC

3) Run knife ec2 server create command using appropriate options.

Following is the command used by us

knife ec2 server create -I ami-d05e75b8 -N <node name> --subnet <vpc subnet-id> -g <security group id inside vpc> --identity-file <somekey.pem> --ssh-user ubuntu --associate-public-ip -c <path to knife.rb> --bootstrap-protocol ssh --flavor <AMI flavor> --server-connect-attribute 'public_ip_address' -VV

Building package using Omnibus

Omnibus is a tool for creating full-stack installers for multiple platforms. In general, it simplifies the installation of any software by including all of the dependencies for that piece of software. It was written by the people at Chef, who use it to package Chef.

Omnibus consists of two pieces- omnibus and omnibus software.

1. omnibus – the framework, created by Chef Software, by which we create full-stack, cross-platform installers for software. The project is on GitHub at chef/omnibus.
2. omnibus-software – Chef Software’s open source collection of software definitions that are used to build the Chef Client, the Chef Server, and other Chef Software products. The software definitions can be found on GitHub at chef/omnibus-software.

Omnibus provides both, a DSL for defining Omnibus projects for your software, as well as a command-line tool for generating installer artifacts from that definition.

Omnibus has minimal prerequisites. It requires Ruby 2.0.0+ and Bundler.

Getting Started

To get started install omnibus :

> gem install omnibus

You can now create an omnibus project inside your current directory using project generator feature

$ omnibus new demo-app

      create  omnibus-demo-app/Gemfile
      create  omnibus-demo-app/.gitignore
      create  omnibus-demo-app/README.md
      create  omnibus-demo-app/omnibus.rb
      create  omnibus-demo-app/config/projects/demo-app.rb
      create  omnibus-demo-app/config/software/demo-app-zlib.rb
      create  omnibus-demo-app/.kitchen.local.yml
      create  omnibus-demo-app/.kitchen.yml
      create  omnibus-demo-app/Berksfile
      create  omnibus-demo-app/package-scripts/demo-app/preinst
       chmod  omnibus-demo-app/package-scripts/demo-app/preinst
      create  omnibus-demo-app/package-scripts/demo-app/prerm
       chmod  omnibus-demo-app/package-scripts/demo-app/prerm
      create  omnibus-demo-app/package-scripts/demo-app/postinst
       chmod  omnibus-demo-app/package-scripts/demo-app/postinst
      create  omnibus-demo-app/package-scripts/demo-app/postrm
       chmod  omnibus-demo-app/package-scripts/demo-app/postrm

It creates the omnibus-demo directory inside your current directory and this directory has all omnibus package build related files. It is easy to build an empty project without doing any change run

$ bundle install --binstubs

bundle install installs all Omnibus dependencies

$ bin/omnibus build demo

The above command will create the installer inside pkg directory. Omnibus determines the platform for which to build an installer based on the platform it is currently running on. That is, you can only generate a .deb file on a Debian-based system. To alleviate this caveat, the generated project includes a Test Kitchen setup suitable for generating a series of Omnibus projects.

Back to the Omnibus DSL. Though bin/omnibus build demo will build the package for you, it will not do anything exciting. For that, you need to use the Omnibus DSL to define the specifics of your application.

1) Config

If present, Omnibus will use a top-level configuration file name omnibus.rb at the root of your repository. This file is loaded at runtime and includes number of configurations.  For e.g.-

# Build locally (instead of /var)# ------------------------------- base_dir './local'  # Disable git caching # ------------------------------ use_git_caching false  # Enable S3 asset caching # ------------------------------ use_s3_caching true s3_access_key  ENV['S3_ACCESS_KEY'] s3_secret_key  ENV['S3_SECRET_KEY'] s3_bucket      ENV['S3_BUCKET']

Please see config doc for more information. You can use different configuration file by using –config option using command line

$ bin/omnibus --config /path/to/config.rb

2) Project DSL

When you create an omnibus project, it creates a project DSL file inside config/project with the name which you used for creating project for above example it will create config/project/demo.rb. It provides means to define the dependencies of the project and metadata of the project.  We will look at some contents of project DSL file

name            "demo" maintainer      "YOUR NAME" homepage        "http://yoursite.com" install_dir     "/opt/demo" build_version   "0.1.0" # Creates required build directories  dependency "preparation"  # demo dependencies/components  dependency "gem name"

‘install_dir’  option is the location where package will be installed. There are more DSL methods available which you can use in this file. Please see the Project Doc for more information.

3) Software DSL

Software DSL defines individual software components that go into making your overall package.  The Software DSL provides a way to define where to retrieve the software sources, how to build them, and what dependencies they have. Now let’s edit a config/software/demo.rb

name "demo" default_version "1.0.0" dependency "ruby" dependency "rubygems" build do    #vendor the gems required by the app    bundle “install –path vendor/bundle” end

In the above example, consider that we are building a package for Ruby on Rails application, hence we need to include ruby and rubygems dependency. The definition for ruby and rubygems dependency comes from theomnibus-software. Chef has introduced omnibus-software, it is a collection of software definitions used by chef while building their products. To use omnibus-software definitions you need to include the repo path in Gemfile. You can also write your own software definitions. Inside build block you can define how to build your installer. Omnibus provide Build DSL which you can use inside build block to define your build essentials. You can run ruby script and copy and delete files using Build DSL inside build block.

Apart from all these DSL file omnibus also created ‘package-script’ directory which consist of pre install and post install script files. You can write a script which you want to run before and after the installation of package and also before and after the removal of the package inside these files.

You can use the following references for more examples

https://github.com/chef/omnibus

http://blog.gemnasium.com/post/60091868742/creating-full-stack-installers-with-omnibus

http://forty9ten.com/post/85546561554/omnibus-installer-tutorial

https://www.chef.io/blog/2014/06/30/omnibus-a-look-forward/

Knife EC2 Winodws bootstrap using Ec2 --user-data option:

Assuming you have a basic knowledge about Chef bootstrapping.

Windows instance bootstrapping requires the winrm enabled images. Either you can create such images and create instance using those image and bootstrap it or use --user-data option to configure winrm while ec2 server create in chef.

Using knife-ec2 chef plugin you can achieve creating windows instance and bootstraping it.

For this you just have to write a proper bash powershell script and reference the path of your script file in --user-date option.

You can use following script to enable winrm and add winrm listener and then use knife-ec2 server create command with --user-data option.

users_data.ps1

<powershell>

$user="<username>"

$password="<password>"

net user /add $user $password;

net localgroup Administrators /add $user;

winrm quickconfig -q

winrm create winrm/config/Listener?Address=*+Transport=HTTP

winrm set winrm/config/service '@{AllowUnencrypted="true"}'

netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" profile=public protocol=tcp localport=5985 remoteip=localsubnet new remoteip=any

</powershell>

And run following command to create ec2 instance.

bundle exec knife ec2 server create -N <node-name> -I <windows-image-id> -f m1.small -x <username> -P <password> --ssh-key <ssh-key-name> -c <path-to-knfie,rb> --bootstrap-protocol winrm --user-data 'C:\\users_data.ps1'  -VV

Similar way you can write the Powershell script to add HTTPS Listener.

users_data.ps1

<powershell>
$user="<username>"
$password="<password>"
$vm_name="<node-name>"
net user /add $user $password;
net localgroup Administrators /add $user;
winrm quickconfig -q
New-SelfSignedCertificate -certstorelocation cert:\localmachine\my -dnsname $vm_name
$thumbprint = (Get-ChildItem -Path cert:\localmachine\my | Where-Object {$_.Subject -match "$vm_name"}).Thumbprint;
$create_listener_cmd = "winrm create winrm/config/Listener?Address=*+Transport=HTTPS '@{Hostname=`"$vm_name`";CertificateThumbprint=`"$thumbprint`"}'"
iex $create_listener_cmd
netsh advfirewall firewall add rule name="WinRM HTTPS" protocol=TCP dir=in Localport=5986 remoteport=any action=allow localip=any remoteip=any profile=public enable=yes
</powershell>

Bootstrap with chef-vault options

What is Chef Valut ?

Chef vault is alternative tool which allows you to securely manage passwords, certs, and other secrets in Chef. It was developed as an open source project originally by Nordstrom, but transferred to Chef in 2015. 

Using knife commands you can create and manage chef-valut

knife vault create passwords mysql '{"user":"root", "password" : "mysqlroot"}'  --mode client -c ~/chef-repo/.chef/knife.rb 

Note : by default the --mode is set to solo and your chef-vault databag will not get generated on chef-server so remember to set it to client.

You can get more details about knife vault option here https://github.com/Nordstrom/chef-vault

The data bag is generated on the chef server & each encrypted Chef-Vault data bag contains two items - a data bag containing the encrypted data & another data bag containing the public keys used to do the encryption.  

Following are the different vault options available for knife bootstrap:

--bootstrap-vault-file VAULT_FILE A JSON file with a list of vault(s) and item(s) to be updated

--bootstrap-vault-item VAULT_ITEM A single vault and item to update as "vault:item"

--bootstrap-vault-json VAULT_JSON A JSON string with the vault(s) and item(s) to be updated

Bootstrap examples with knife windows

knife bootstrap windows winrm <FQDN> -x <username> -P <password> -c <knife.rb path> --bootstrap-valut-file <path to valut file>

knife bootstrap windows winrm <FQDN> -x <username> -P <password> -c <knife.rb path> --bootstrap-vault-item '{"passwords": "mysql"}

knife bootstrap windows winrm <FQDN> -x <username> -P <password> -c <knife.rb path>

--bootstrap-vault-json '{"passwords": "mysql", "creds": "fooapi"}'