Essential components make up the self-deploying infrastructure that is the heart of TripleO.
These components add value to the TripleO story, making it safer to upgrade and evolve an environment, but are secondary to the core thing itself.
Each component can only be deployed once its dependencies are available.
TripleO is built on a Linux platform, so a Linux environment is required both to create images and as the OS that will run on the machines. If you have no Linux machines at all, you can download a live CD from a number of vendors, which will permit you to run diskimage-builder to get going.
An internet connection is also required to download the various packages used in preparing each image.
The machine images built can depend on Heat metadata, or they can just contain configured Chef/Puppet/Salt credentials, depending on how much of TripleO is in use. Avoiding Heat is useful when doing a incremental adoption of TripleO (see later in this document).
Baremetal deployments are delivered via Nova. Additionally, the network must be configured so that the baremetal host machine can receive TFTP from any physical machine that is being booted.
Nova depends on Keystone, Glance and Neutron. In future Cinder will be one of the dependencies.
There are three ways the service can be deployed:
Cinder is needed for persistent storage on bare metal machines. That aspect of TripleO is not yet available : when an instance is deleted, the storage is deleted with it.
Neutron depends on Keystone. The same three deployment options exist as for Nova. The Neutron network node(s) must be the only DHCP servers on the network.
Glance depends on Keystone. The same three deployment options exist as for Nova.
Keystone has no external dependencies. The same three deployment options exist as for Nova.
Heat depends on Nova, Cinder and Keystone. The same three deployment options exist as for Nova.
The os-config-applier and os-refresh-config tools depend on Heat to provide cluster configuration metadata. They can be used before Heat is functional if a statically prepared metadata file is placed in the Heat path : this is how the bootstrap node works.
os-config-applier and os-refresh-config can be used in concert with Chef/Puppet/Salt, or not used at all, if you configure your services via Chef/Puppet/Salt.
The reference TripleO elements do not depend on Chef/Puppet/Salt, to avoid conflicting when organisations with an investment in Chef/Puppet/Salt start using TripleO.
The general sequence is:
These are all documented in README.rst and in the TripleO bugtracker.
Every ‘nova boot’ will reset the data on the machine it deploys to. To do incremental image based updates they have to be done within the runnning image. ‘takeovernode’ can do that, but as yet we have not written rules to split out persistent data into another partition - so some assembly required.
If you require VLANs you should create a diskimage-builder element to add the vlan package and vlan configuration to /etc/network/interfaces as a first-boot rule.
Creating a new seed image takes up to 4.5GB of space inside a /tmp/imageXXXXX directory. tmpfs can take up to 50% of RAM and systems with less than 9GB of RAM will fail in this step. When using diskimage-builder directly, you can prevent the space errors by:
If you are using boot-seed-vm, set the environment variable DIB_NO_TMPFS=1.
In this scenario you make use of the baremetal driver to deploy unspecialised machine images, and perform specialisation using Chef/Puppet/Salt - whatever configuration management toolchain you prefer. The baremetal host system is installed manually, but a TripleO image is used to deploy it.
It scales within any one broadcast domain to the capacity of the single baremetal host.
Build the images you need (add any local elements you need to the commands)
Copy tripleo-image-elements/elements/seed-stack-config/config.json to tripleo-image-elements/elements/seed-stack-config/local.json and change the virtual power manager to ‘nova...impi.IPMI’. https://bugs.launchpad.net/tripleo/+bug/1178547:
disk-image-create -o bootstrap vm boot-stack local-config ubuntu
disk-image-create -o ubuntu ubuntu
The local-config element will copy your ssh key and your HTTP proxy settings in the disk image during the creation process.
The stackuser element will create a user stack with the password stack.
disk-image-create will create a image with a very small disk size that at to be resized for example by cloud-init. You can use DIB_IMAGE_SIZE to increase this initial size, in GB.
Setup a VM using bootstrap.qcow2 on your existing machine, with eth1 bridged into your datacentre LAN.
Run up that VM, which will create a self contained nova baremetal install.
Reconfigure the networking within the VM to match your physical network. https://bugs.launchpad.net/tripleo/+bug/1178397 https://bugs.launchpad.net/tripleo/+bug/1178099
If you had exotic hardware needs, replace the deploy images that the bootstack creates. https://bugs.launchpad.net/tripleo/+bug/1178094
Enroll your vanilla image into the glance of that install. Be sure to use tripleo-incubator/scripts/load-image as that will extract the kernel and ramdisk and register them appropriately with glance.
Enroll your other datacentre machines into that nova baremetal install. A script that takes your machine inventory and prints out something like:
nova baremetal-node-create --pm_user XXX --pm_address YYY --pm_password ZZZ COMPUTEHOST 24 98304 2048 MAC
can be a great help - and can be run from outside the environment.
Setup admin users with SSH keypairs etc. e.g.:
nova keypair-add --pub-key .ssh/authorized_keys default
Boot them using the ubuntu.qcow2 image, with appropriate user data to connect to your Chef/Puppet/Salt environments.
In this scenario you use the baremetal driver to deploy specialised machine images which are orchestrated by Heat.
Build the images you need (add any local elements you need to the commands):
disk-image-create -o bootstrap vm boot-stack ubuntu heat-api
disk-image-create -o ubuntu ubuntu cfn-tools
Setup a VM using bootstrap.qcow2 on your existing machine, with eth1 bridged into your datacentre LAN.
Run up that VM, which will create a self contained nova baremetal install.
Enroll your vanilla image into the glance of that install.
Enroll your other datacentre machines into that nova baremetal install.
Setup admin users with SSH keypairs etc.
Create a Heat stack with your application topology. Be sure to use the image id of your cfn-tools customised image.
In this scenario we build on Baremetal with Heat to deploy a full OpenStack orchestrated by Heat, with specialised disk images for different OpenStack node roles.
Follow the ‘devtest’ guide but edit the seed config.json to:
change the dnsmasq range to the seed deployment range
change the heat endpoint details to refer to your seed ip address
change the ctlplane ip and cidr to match your seed ip address
change the power manager line nova.virt.baremetal.ipmi.IPMI and remove the virtual subsection.
register the undercloud machine with its details rather than generic virtual ones - e.g:
setup-baremetal 24 98304 1500 amd64 $somemac undercloud $ipmi_ip $ipmi_user $ipmi_password
setup proxy arp (this and the related bits are used to avoid messing about with the public NIC and bridging: you may choose to use that approach instead...):
sudo sysctl net/ipv4/conf/all/proxy_arp=1
arp -s <seedip> -i <external_interface> -D <external_interface> pub
ip addr add <seediplink>/32 dev brbm
ip route add <seedip>/32 dev brbm src <seediplink>
setup ec2 metadata support:
iptables -t nat -A PREROUTING -d 169.254.169.254/32 -i <external_interface> -p tcp -m tcp --dport 80 -j DNAT --to-destination <seedip>:8775
setup DHCP relay:
sudo apt-get install dhcp-helper
and configure it with -s <seedip> Note that isc-dhcp-relay fails to forward responses correctly, so dhcp-helper is preferred ( https://bugs.launchpad.net/ubuntu/+bug/1233953 ).
Also note that dnsmasq may have to be stopped as they both listen to *:dhcps ( https://bugs.launchpad.net/ubuntu/+bug/1233954 ).
Disable the filter-bootps cronjob (./etc/cron.d/filter-bootp) inside the seed vm and reset the table:
sudo iptables -F FILTERBOOTPS
edit /etc/init/novabm-dnsmasq.conf:
exec dnsmasq --conf-file= \
--keep-in-foreground \
--port=0 \
--dhcp-boot=pxelinux.0,<seedip>,<seedip> \
--bind-interfaces \
--pid-file=/var/run/dnsmasq.pid \
--interface=br-ctlplane \
--dhcp-range=<seed_deploy_start>,<seed_deploy_end>,<network_cidr>
When you setup the seed, use <seedip> instead of 192.0.2.1, and you may need to edit seedrc.
For setup-neutron: setup-neutron <start of seed deployment> <end of seed deployment> <cidr of network> <seedip> <metadata server> ctlplane
Validate networking:
Create your deployment ramdisk with baremetal in mind:
$TRIPLEO_ROOT/diskimage-builder/bin/disk-image-create $NODE_DIST -a \
$NODE_ARCH -o $TRIPLEO_ROOT/undercloud boot-stack nova-baremetal \
os-collect-config stackuser $DHCP_DRIVER -p linux-image-generic mellanox \
serial-console --offline
If your hardware has something other than eth0 plugged into the network, fix your file injection template - /opt/stack/nova/nova/virt/baremetal/net-static.ubuntu.template inside the seed vm, replacing the enumerated interface values with the right interface to use (e.g. auto eth2... iface eth2 inet static..)
Use heat stack-create per the devtest documentation to boot your undercloud. But use the undercloud-bm.yaml file rather than undercloud-vm.yaml.
Once it’s booted:
modprobe 8021q
edit /etc/network/interfaces and define your vlan
delete the default route on your internal network
add a targeted route to your management l3 range via the internal network router
add a targeted route to 169.254.169.254 via <seedip>
ifup the vlan interface
fix your resolv.conf
configure the undercloud per devtest.
upgrade your quotas:
nova quota-update --cores node_size*machine_count --instances machine_count --ram node_size*machine_count admin-tenant-id
Follow devtest again, but modify the images you build per the undercloud notes, and for machines you put public services on, follow the undercloud notes to fix them up.
WARNING: Here be draft notes.