Puppet is a cross-platform system for managing machines. It can manage files, packages, users and anything else that needs managing. The system is composed of a puppetd daemon process running on a client machine, which periodically connects to a remote puppetmaster to download its configuration and apply it. All communication and file transfers between the client and puppetmaster are done via XML/RPC over SSL.

If you'd like to know more about what else puppet can do or how it works, reductivelabs has an excellent introduction. This article will instead focus on how to set up an environment for measuring the throughput of the puppetmaster.

This exercise is best done with a fully functioning puppet system, meaning that you'll need access to a puppetmaster server and a working client. You'll need to install packages for openssl and ssldump, which are available in most distributions.

We'll be executing our benchmarks with jmeter, a popular socket-based benchmarking suite from the apache jakarta project. Before we even open the application, we should address two issues that complicate benchmarking:

1. Puppet systems make use of a CA to issue SSL certificates and to authenticate communication between clients and servers. We'll need the puppet CA to issue a certificate to our jmeter machine if it doesn't already have one.
   
2. When the client requests its catalog from the server, it provides the server information about itself via facter. The server uses this information when it compiles the clients catalog. Our test suite will need dummy data in the correct format when it requests its catalog from the puppetmaster

Neither of these are insurmountable. Let's get started.

In my test setup I've chosen to run jmeter on a non-puppet machine. If you are running jmeter on a machine that already has puppet installed then you should use the certificate-key pair previously issued by the puppet CA.

Without pre-existing keys and certs, we'll need to generate them on the CA and copy them to our testing machine. By default, the CA is the same machine as the puppetmaster, though most large setups will have a dedicated CA that is set apart from the puppetmaster.

JMeter has only a limited understanding of the different certificate types out there, so we'll need to remux our key and cert into a PKCS12 cert that it can use. We also need to set a password along the way, which is less of a security consideration than it is a workaround for a bug in the java security libraries. It should be at least six characters long.

After setting the password, you should have a p12 certificate file usable in jmeter.

Of the different calls that a puppet client can invoke on the puppetmaster, the one that we're most interested in is puppetmaster.getconfig. This call is made on every puppet run and returns a compiled catalog to the client. It requires the puppetmaster to resolve a graph of resource dependencies and is generally the most intensive task that a puppetmaster will do.

When the client makes the getconfig call, it submits a url-encoded string of fact names and values generated by facter. We'll need genuine facter data to feed to the puppetmaster if we're expecting a realistic test. We can get this data by using ssldump, an analyzer capable of decrypting SSL traffic. Decryption is done using the private key of the server, so we'll want to run this on our puppetmaster:

We can then invoke a puppet run on a representative puppet client. I chose to use one of my ldap servers.

After the puppet client has finished running, we can start sifting through the traffic dump. We should have a couple regions of decrypted requests and responses, but I've found that there are some setups that cannot be decrypted by ssldump. If the last lines printed by ssldump look like these, then your data was not decrypted.

You could refer to the ssldump troubleshooting document (see PROBLEM 2: decryption doesn't work), or try this alternate method of generating request data.

You can learn a lot about how puppet works just by examining the traffic between client and server, but the part we're interested in is this:

We'll want to paste the headers and body of this call into a file somewhere on our jmeter machine. Before proceeding any further, we should stop and test our ability to connect to the puppetmaster with our generated certificate, make a fake getconfig call, and receive a meaningful response from our test subject. We can do this on our jmeter machine with openssl

From here we can paste the headers and request body of the call that we saved earlier. Make sure that the headers and request body are separated by a blank line and that the request body is followed by a newline character. The server should quickly (or slowly) return a response that looks like:

Looks like we're in business! It's time to start building our test plan in JMeter.

You'll need to install jmeter if you don't already have it. This article was written for jmeter 2.3.2, though other versions should be very similar if not identical. We'll walk through constructing a simple load test using the request we hijacked earlier.

On first opening jmeter, you should be presented with a plain workspace.

We'll start by adding a Thread Group to the project through the Test Plans right-click menu. For now, the thread group should be limited to 1 thread and 1 loop.

To the thread group we'll add a SOAP/XML-RPC sampler, which we'll configure such that:

• URL points to https://<puppetmaster>:<port>/RPC2
  
• Send SOAPAction is unchecked
  
• Use KeepAlive is checked
  
• Soap/XML-RPC data contains the request we obtained earlier, sans headers

We'll finish off our test plan by adding some listener elements to the sampler, letting us see the server responses as the test threads run. Finally, before we can begin executing our test, we need to point jmeter to the certificate we created earlier. Without it, the puppetmaster will consider our test requests to be unauthorized and they will be summarily dropped.

We can add certificates through the jmeter SSL Manager (in the Options menu). We'll use the finder to open the PKCS12 cert we created earlier.

After that, we can start our first run. JMeter will want you to save your test plan first, and we'll have to give the password to the certificate created earlier (if jmeter doesn't ask you for a password then chances are good that your password was not long enough and that your certificate has been ignored).

The test should run quickly and if you've created a 'View Results in Table' listener earlier, your efforts will be rewarded with a green check mark. Now that we're all set up and jmeter knows our cert password, we can go back and begin tweaking the test. I've gone back and boosted the number of threads and iterations in the thread group, which generated this throughput chart

We should be careful to understand what we have and have not tested here. Our chart gives a fair assessment of the throughput of a puppetmaster, as well as the performance degradation that occurs when bringing up new nodes (modeled here as jmeter threads). We can also see what happens on the puppetmaster when it is flooded with requests, and using different facts and samplers we can simulate both intensive and non-intensive catalog requests.

Perhaps more important are the things not tested. This setup will tell us nothing about report processing, nor tell us anything about the memory issues related to file serving. These tests only give us timings from the clients perspective, and cannot separate the time the server requires to compile the catalog from the time it takes to send the compiled result to the client. We'll explore more of these server side issues in a few weeks with part 2 of this series.