From 47cef23099f8d68c451792bb50730a5d324eacdb Mon Sep 17 00:00:00 2001 From: Chris Price Date: Tue, 18 Sep 2012 14:32:34 -0700 Subject: [PATCH] Add initial READMEs and CHANGELOG --- CHANGELOG | 2 + README.md | 74 +++++++++- README_GETTING_STARTED.md | 279 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 353 insertions(+), 2 deletions(-) create mode 100644 CHANGELOG create mode 100644 README_GETTING_STARTED.md diff --git a/CHANGELOG b/CHANGELOG new file mode 100644 index 0000000..97d15c7 --- /dev/null +++ b/CHANGELOG @@ -0,0 +1,2 @@ +2011-09-18 Chris Price - 1.0.0 +* Initial 1.0.0 release diff --git a/README.md b/README.md index eacf836..d27d8d7 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,76 @@ puppetlabs-puppetdb =================== -A puppet module for installing and managing puppetdb +Purpose: Install and manage the PuppetDB server and database, and + configure the Puppet master to use PuppetDB +Module: puppetlabs/puppetdb (http://forge.puppetlabs.com/cprice404/puppetdb) +Puppet Version: 2.7+ +Platforms: RHEL6, Debian6, Ubuntu 10.04 -#TODO: more docs +Installing and configuring PuppetDB isn’t *too* difficult, but we knew that it +could and should be even easier than it was. That’s where the new +`puppetlabs/puppetdb` module comes in. Whether you just want to throw PuppetDB +onto a test system as quickly as possible so that you can check it out, or you +want finer-grained access to managing the individual settings and configuration, +this module aims to let you dive in at exactly the level of involvement that you +desire. + +Here are some of the capabilities of the module; almost all of these are optional, +so you are free to pick and choose which ones suit your needs: + +* Installs and manages the core PuppetDB server +* Installs and manages the underlying database server (PostgreSQL or a simple +embedded database) +* Configures your Puppet master to use PuppetDB +* Optional support for opening the PuppetDB port in your firewall on +RedHat-based distros +* Validates your database connection before applying PuppetDB configuration +changes, to help make sure that PuppetDB doesn’t end up in a broken state +* Validates your PuppetDB connection before applying configuration changes to +the Puppet master, to help make sure that your master doesn’t end up in a broken +state + +Resource Overview +----------------- + +##### `puppetdb` class + +This is a sort of ‘all-in-one’ class for the PuppetDB server. It’ll get you up +and running with everything you need (including database setup and management) +on the server side. The only other thing you’ll need to do is to configure your +Puppet master to use PuppetDB... which leads us to: + +##### `puppetdb::master::config` class + +This class should be used on your Puppet master node. It’ll verify that it can +successfully communicate with your PuppetDB server, and then configure your +master to use PuppetDB. + +***NOTE***: Using this class involves allowing the module to manipulate your +puppet configuration files; in particular: `puppet.conf` and `routes.yaml`. The +`puppet.conf` changes are supplemental and should not affect any of your existing +settings, but the `routes.yaml` file will be overwritten entirely. If you have an +existing `routes.yaml` file, you will want to take care to use the `manage_routes` +parameter of this class to prevent the module from managing that file, and +you’ll need to manage it yourself. + +##### `puppetdb::server` class + +This is for managing the PuppetDB server independently of the underlying +database that it depends on; so it’ll manage the PuppetDB package, service, +config files, etc., but will allow you to manage the database (e.g. postgresql) +however you see fit. + +###### `puppetdb::database::postgresql` class + +This is a class for managing a postgresql server for use by PuppetDB. It can +manage the postgresql packages and service, as well as creating and managing the +puppetdb database and database user accounts. + +##### Low-level classes + +There are several lower-level classes in the module (e.g., `puppetdb::master::*` +and `puppetdb::server::*` which you can use to manage individual configuration +files or other parts of the system. In the interest of brevity, we’ll skip over +those for now... but if you need more fine-grained control over your setup, feel +free to dive into the module and have a look!) diff --git a/README_GETTING_STARTED.md b/README_GETTING_STARTED.md new file mode 100644 index 0000000..0cf39c8 --- /dev/null +++ b/README_GETTING_STARTED.md @@ -0,0 +1,279 @@ +puppetlabs/puppetdb - PuppetDB Management +----------------------------------------- + +Purpose: Install and manage the PuppetDB server and database, and + configure the Puppet master to use PuppetDB +Module: puppetlabs/puppetdb (http://forge.puppetlabs.com/cprice404/puppetdb) +Puppet Version: 2.7+ +Platforms: RHEL6, Debian6, Ubuntu 10.04 + +One of the new projects that we at Puppet Labs are excited about right now is +PuppetDB, our new “data warehouse” for managing storage and retrieval of all +platform-generated data. (If you haven’t checked it out yet, have a look at +[Nick Lewis’ blog +post](http://puppetlabs.com/blog/introducing-puppetdb-put-your-data-to-work/) or +the [PuppetDB documentation](http://docs.puppetlabs.com/puppetdb/).) Currently, +it offers a huge performance improvement for exported and collected resources, +as well as several other great features. We’re even more excited about some of +the not-quite-released functionality that is in the pipeline, so stay tuned for +more information! + +Installing and configuring PuppetDB isn’t *too* difficult, but we knew that it +could and should be even easier than it was. That’s where the new +`puppetlabs/puppetdb` module comes in. Whether you just want to throw PuppetDB +onto a test system as quickly as possible so that you can check it out, or you +want finer-grained access to managing the individual settings and configuration, +this module aims to let you dive in at exactly the level of involvement that you +desire. + +Here are some of the capabilities of the new 1.0 release of the `puppetdb` +module; almost all of these are optional, so you are free to pick and choose +which ones suit your needs: + +* Installs and manages the core PuppetDB server +* Installs and manages the underlying database server (PostgreSQL or a simple +embedded database) +* Configures your Puppet master to use PuppetDB +* Optional support for opening the PuppetDB port in your firewall on +RedHat-based distros +* Validates your database connection before applying PuppetDB configuration +changes, to help make sure that PuppetDB doesn’t end up in a broken state +* Validates your PuppetDB connection before applying configuration changes to +the Puppet master, to help make sure that your master doesn’t end up in a broken +state + +Installing the module +--------------------- + +Installing the PuppetDB module is a breeze using the Puppet module tool +(available in Puppet 2.7.14+ and Puppet Enterprise 2.5+): + + $ puppet module install puppetlabs/puppetdb + puppet module install puppetlabs/puppetdb + Preparing to install into /etc/puppet/modules ... + Downloading from http://forge.puppetlabs.com ... + Installing -- do not interrupt ... + /etc/puppet/modules + └─┬ puppetlabs-puppetdb (v0.1.1) + ├── cprice404-inifile (v0.0.2) + ├─┬ inkling-postgresql (v0.3.0) + │ └── puppetlabs-stdlib (v3.0.1) + └── puppetlabs-firewall (v0.0.4) + $ + +Resource Overview +----------------- + +Let’s take a quick peek at the main classes and types defined by the module. +(We’ll take a more in-depth look, with examples, in the following section.) + +##### `puppetdb` class + +This is a sort of ‘all-in-one’ class for the PuppetDB server. It’ll get you up +and running with everything you need (including database setup and management) +on the server side. The only other thing you’ll need to do is to configure your +Puppet master to use PuppetDB... which leads us to: + +##### `puppetdb::master::config` class + +This class should be used on your Puppet master node. It’ll verify that it can +successfully communicate with your PuppetDB server, and then configure your +master to use PuppetDB. + +***NOTE***: Using this class involves allowing the module to manipulate your +puppet configuration files; in particular: `puppet.conf` and `routes.yaml`. The +`puppet.conf` changes are supplemental and should not affect any of your existing +settings, but the `routes.yaml` file will be overwritten entirely. If you have an +existing `routes.yaml` file, you will want to take care to use the `manage_routes` +parameter of this class to prevent the module from managing that file, and +you’ll need to manage it yourself. + +##### `puppetdb::server` class + +This is for managing the PuppetDB server independently of the underlying +database that it depends on; so it’ll manage the PuppetDB package, service, +config files, etc., but will allow you to manage the database (e.g. postgresql) +however you see fit. + +###### `puppetdb::database::postgresql` class + +This is a class for managing a postgresql server for use by PuppetDB. It can +manage the postgresql packages and service, as well as creating and managing the +puppetdb database and database user accounts. + +##### Low-level classes + +There are several lower-level classes in the module (e.g., `puppetdb::master::*` +and `puppetdb::server::*` which you can use to manage individual configuration +files or other parts of the system. In the interest of brevity, we’ll skip over +those for now... but if you need more fine-grained control over your setup, feel +free to dive into the module and have a look!) + +Example Usage +------------- + +Enough with the gory details, let’s talk about how to actually use the thing! + +When you are first getting started with PuppetDB, there are a few decision +you’ll have to make: + +* Which database back-end should I use? (The current choices are PostgreSQL or +our embedded database; we’ll discuss this more a bit later on.) +* Should I run the database on the same node that I run PuppetDB on? +* Should I run PuppetDB on the same node that I run my master on? + +The answers to those questions will be largely dependent on your Puppet +environment. How many nodes are you managing? What kind of hardware are you +running on? Is your current load approaching the limits of your hardware? + +### The Simple Case + +Since I won’t be able to answer all of those questions for you, we’ll start off +with the absolute simplest case: using our default database (PostgreSQL), and +running everything (PostgreSQL, PuppetDB, Puppet master) all on the same node. +This setup will be great for testing / experimental environment, and may be +sufficient for many real-world deployments depending on the number of nodes +you’re managing. So, what would our manifest look like in this case? + + node puppetmaster { + # Configure puppetdb and its underlying database + include puppetdb + # Configure the puppet master to use puppetdb + include puppetdb::master::config + } + + +That’s it! Obviously, you can provide some parameters for these classes if +you’d like more control, but that is literally all that it will take to get you +up and running with the default configuration. Here are the steps that this +manifest will trigger: + +* Install PostgreSQL on the node if it’s not already there +* Create the PuppetDB postgres database instance and user account +* Validate the postgres connection and, if successful, install and configure +PuppetDB +* Validate the PuppetDB connection and, if successful, modify the Puppet master +config files to use PuppetDB + +You will need to restart your puppet master one time after the run, in order for +all of these changes to take effect. If your logging level is set to INFO or +finer, you should start seeing PuppetDB-related log messages appear in both your +Puppet master log and your PuppetDB log as subsequent agent runs occur. + +Note: If you’d prefer to use PuppetDB’s embedded database rather than +PostgreSQL, have a look at the database parameter on the puppetdb class. The +embedded db can be useful for testing and very small production environments, +but is not recommended for production environments as it consumes a great deal +of memory as your number of nodes increases. + +### A Distributed Setup + +In many cases, you’ll prefer not to install PuppetDB on the same node as the +Puppet master. Your environment will be easier to scale if you are able to +dedicate hardware to the individual system components. You may even choose to +run the PuppetDB server on a different node from the PostgreSQL database that it +uses to store its data. So let’s have a look at what a manifest for that +scenario might look like: + + # This is an example of a very basic 3-node setup for PuppetDB. + + # This node is our Puppet master. + node puppet { + # Here we configure the puppet master to use PuppetDB, + # and tell it that the hostname is ‘puppetdb’ + class { 'puppetdb::master::config': + puppetdb_server => 'puppetdb', + } + } + + # This node is our postgres server + node puppetdb-postgres { + # Here we install and configure postgres and the puppetdb + # database instance, and tell postgres that it should + # listen for connections to the hostname ‘puppetdb-postgres’ + class { 'puppetdb::database::postgresql': + listen_addresses => 'puppetdb-postgres', + } + } + + # This node is our main puppetdb server + node puppetdb { + # Here we install and configure PuppetDB, and tell it where to + # find the postgres database. + class { 'puppetdb::server': + database_host => 'puppetdb-postgres', + } + } + +That’s it! This should be all it takes to get a 3-node, distributed +installation of PuppetDB up and running. Note that if you prefer, you could +easily move two of these classes to a single node and end up with a 2-node setup +instead. + +### Cross-node Dependencies + +If you’re playing along at home, you may have spotted some cross-node +dependencies here and you’ve probably recognized that the order that these nodes +check in with the puppet master will have serious implications for getting +everything up and running. It would be very bad to configure the master to use +the PuppetDB server before that server was up and running. Likewise, it +wouldn’t be great to try to start up the PuppetDB server pointing to a Postgres +server that isn’t actually running Postgres yet. + +The module handles this problem for you by taking a sort of “eventual +consistency” approach. There’s nothing that the module can do to control the +order in which your nodes check in, but the module *can* check to verify that +the services it depends on are up and running before it makes configuration +changes--so that’s what it does. + +When your Puppet master node checks in, it will validate the connectivity to the +PuppetDB server before it applies its changes to the Puppet master config files. + If it can’t connect to PuppetDB, then the puppet run will fail and the previous +config files will be left intact. This prevents your master from getting into a +broken state where all incoming Puppet runs fail because the master is +configured to use a PuppetDB server that doesn’t exist yet. The same strategy +is used to handle the dependency between the PuppetDB server and the postgres +server. + +What does this all mean to you, as a user? Well, it basically means that the +first time you add this stuff to your manifests, you may see a few failed Puppet +runs on the affected nodes. This should be limited to 1 failed run on the +PuppetDB node, and up to 2 failed runs on the Puppet master node. After that, +all of the dependencies should be satisfied and your puppet runs should start to +succeed again. + +If you prefer, you can manually trigger puppet runs on the nodes in the correct +order (Postgres, PuppetDB, Puppet master) and you should avoid any failed runs. + +Configuring the module +---------------------- + +The module supports a large number of configuration options. If you’d like more +control over things like: + +* whether or not to open the PuppetDB port on the firewall +* what address the PuppetDB server should listen on +* what version of PuppetDB to use +* what address the PostgreSQL server should listen on +* PostgreSQL database name, username, password, etc. +* custom paths to various configuration files + +and more, please take a peek at the individual classes. They expose a large +number of parameters and should hopefully be documented fairly well. (We won’t +cover them here since this post has already gotten a bit long-winded, if I do +say so myself, but perhaps we’ll do a follow-up blog post in the future that +goes into greater detail.) + +Conclusion +---------- + +That’s about it for now. We hope that this module makes it So Darn Easy to get +up and running with PuppetDB that you simply can’t come up with any more excuses +not to go ahead and do it right now! We think you’ll be happy you did--not only +because of its current power and features, but also because of all of the great +things we have in store for it in the near future. + +If you have any questions, suggestions, or feedback, please send them to Ryan +or Chris! If there’s a setting that you’d like to be able to manage that we +haven’t exposed yet, let us know, or better yet, file a pull request to the +module project: https://github.com/puppetlabs/puppetlabs-puppetdb