Proof reading and helpful contributions: Joseph Mack
Copyright © 2003 Jeremy Kerr
v2003.07 Jul 2003, released under GPL.
Table of Contents
- 1. Introduction
- 2. Configure LVS and network
- 3. Deploying the master
- 3.1. Download and untar
- 3.2. Configure
- 3.3. Make
- 3.4. Install
- 3.5. Edit master configuration file
- 3.6. Running feedbackd-master
- 4. Deploying the agent
- 4.1. Download and untar
- 4.2. Configure
- 4.3. Make
- 4.4. Install
- 4.5. Edit agent configuration file
- 4.6. Running feedbackd-agent
- 4.7. Monitor Plugins
- 5. Security
- 6. Adjusting feedbackd parameters
Abstract
This document is provided to assist in deploying the feedbackd system on a Linux Virtual Server (LVS) based server cluster. It is assumed that the LVS system is working, and you have a basic understanding of Linux and compiling software packages.
The LVS HOWTO gives a comprehensive guide to setting up an LVS cluster.
The feedbackd system is aimed at improving the performance of a server cluster, by evenly distributing a request load amongst the available servers. It does this by reporting the servers' load to the director, where it is used to allocate subsequent requests to the least busy servers.
To do this, feedbackd is split into two processes - the master, which is run on the director, and the agents, which are run on the servers. The agents and master communicate using the Network Element Control Protocol (NECP).
The following terms are used in this document:
- cluster
- The set of nodes that provide the entire load-balanced network service. This includes director(s) and realservers.
- director
- The host performing the load balancing, using the LVS software. Requests (from the clients) are first received by a director and forwarded to a realserver to be processed.
- realserver
- The host providing the network service.
- realserver health
- A metric of a realserver's ability to provide a network service. There are a number of possible measurements for this (for example, cpu load). The most suitable measure will depend on the particular network service that the realserver provides.
- realserver weight
- The director must allocate incoming requests to a particular realserver. The weight of each realserver is the probability that it will be chosen to process the request.
- master
- The feedbackd process that runs on the director. The master's role is to request health information from the agents and update the weight for each realserver accordingly.
- agent
- The feedbackd process that runs on the realservers. Each agent connects to the master and reports realserver health when polled.
When a command is to be entered to the shell, a prompt is given (for example director:$), specifying the host to enter the command on (either director or realserver). The host is followed by either a $ (indicating that the command is to be entered as a normal user) or a # (indicating that the command is to be entered by the root user).
This document was published using the following versions of the feedbackd software:
| feedbackd-master | 0.3.1 |
| feedbackd-agent | 0.3.1 |
However, most of the information contained should be relevant to earlier versions
Feedbackd uses the Network Element Control Protocol (NECP) to communicate load data between master and agents. The NECP specification is available at http://www.circlemud.org/~jelson/writings/draft-cerpa-necp-03.txt. An overview of NECP is available at http://www.circlemud.org/~jelson/writings/necp-ietf/.
A typical NECP session proceeds like this:
- The master starts, and listens for connections on 3262
- An agent starts, connects to the master and performs initialisation
- The agent informs the master of the services that it can provide. This is defined in the configuration file, typically /etc/feedbackd-agent.xml
- The master registers these services in the ipvs tables, initially with a "weight" of zero, so no requests will be forwarded to the realserver
- Periodically, the master will ask the agent for its current health status for each of its services. When the reply is received by the master, it updates the ipvs tables with the new data. The NECP specification calls a request for health data a 'KEEPALIVE request'. The reply (which contains the health data) is called a 'KEEPALIVE ACK'.
- If no response is seen to a keepalive packet, the master resends it. After a predefined number of retries, the realserver is considered dead and quiesced.
- When an agent is shutdown, it tells the master, which queisces the service (by setting the realserver's weight to zero again) - no new connections will be made to the realserver
- For any services in the process of quiescing, the master checks the number of active connections to that service on a regular basis. When this number of connections is zero, the service is removed from the ipvs tables.
This procedure outlines a few important concepts:
- The agents are responsible for finding the master process - so must be configured with the address of the director node. The director has no advance knowlegde of where its realservers will be, or the forwarding type required to send requests to the realserver.
- The realserver contacts the director with a list of the services it has available for the LVS.
- These previous two points allow some nice functionality - no reconfiguration needs to be done to the director when new realservers are added to the cluster. New realservers can be added by plugging them into the network and running the agent. Similarly, to remove a realserver, the agent is halted - after all existing connections have been processed, the realserver can be removed.
To deploy feedbackd, you must have a working LVS-based cluster. Any type of forwarding type will work, as long as your network topology allows the realservers to establish a tcp session to port 3262 on the director.
The master process is installed on the director - its role is to listen for connections from the agents, and update the director's ipvs tables with health values reported by the agents
The master requires the libxml2 library to parse configuration data. If installing from a package-management system (for example, .rpm or .deb), you'll need the libxml2 development package.
Generally, these are the steps required (the arguments to configure are only a guide):
director:$ tar zxvf feedbackd-master-0.3.1.tar.gz director:$ cd feedbackd-master-0.3.1 director:$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var director:$ make director:$ su director:# make install director:# $EDITOR /etc/feedbackd-master.xml director:# feedbackd-master -t director:# feedbackd-master |
The feedbackd-master source distribution is available from http://www.redfishsoftware.com.au/projects/feedbackd/. Untar the files into a build directory:
director:$ tar zxvf feedbackd-master-0.3.1.tar.gz director:$ cd feedbackd-master-0.3.1 |
The feedbackd distribution is built using the autoconf and automake tools, so configuration is fairly standard:
director:$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var |
Feel free to use other values for the prefix or sysconfdir paths, although this document assumes these options are used - you'll need to change the paths of subsequent commands.
Options to configure:
- --without-ipvs
- Make a master binary that does not interface to the ipvs system. This is useful for testing on a system that does not have the ipvs kernel or libipvs installed.
- --with-linux-dir=PATH
- Where to find the Linux includes. Defaults to /usr/src/linux
- --with-xml-config=PATH
- Where to find the xml-config or xml2-config program. This is required to find the appropriate compiler flags and libraries to use for XML parsing.
If configure completes without errors, skip to the next section
Otherwise, if you get the following error:
configure: error: cannot find net/ip_vs.h in your kernel source. |
then the ipvs headers are not installed in the expected location. There are a few possible reasons:
- The kernel has not been patched with the ipvs software. Either patch and recompile the kernel, or if you're testing feedbackd on a non-LVS kernel, configure feedbackd with --without-ipvs
- The kernel headers are not in /usr/src/linux. Add a --with-linux-dir=PATH-TO-LINUX-DIR argument to configure (for example, --with-linux-dir=/usr/src/linux-2.4.21)
If you get the following error:
configure: error: cannot find ipvs/ipvs.h. Is ipvsadm installed? |
or
configure: error: cannot find libipvs library. Is ipvsadm installed? |
then ipvsadm has not installed its header files and/or library in an accessible location. If you have not installed ipvsadm, do that now. Otherwise, you have two options here (where $ipvsadm_root is where you have extracted the ipvsadm tarball):
- Apply the patch included with feedbackd-master, ipvsadm-1.21.libinstall.diff to ipvsadm, and recompile it:
director:$ cd $ipvsadm_root director:$ patch -p1 < /path/to/feedbackd-master-0.3.1/ipvsadm-1.21.libinstall.diff director:$ make director:$ su director:# make install
- Copy the header and include files manually:
director:$ cd $ipvsadm_root director:$ su director:# cp libipvs/libipvs.a /usr/lib director:# mkdir /usr/include/ipvs director:# cp libipvs/libipvs.h /usr/include/ipvs/ipvs.h
Now you can build:
director:$ make |
If make fails, send me an email. Include as much detail as possible (including errors returned by make, and your config.* files)
.. and install - you may have to su to root to do this, depending on the --prefix passed to configure.
director:$ su director:# make install |
This will install the following files:
- The feedbackd-master executable
- The configuration file, feedbackd-master.xml
- The man page
 | Note | |
|---|---|---|
The --prefix argument (given to ./configure) is used in various stages of building, so changing the prefix in the install stage will not work. In short, do not specify an different prefix in the install stage:
| ||
The master configuration file is installed in /etc/feedbackd-master.xml by default. It's in XML format, and is fairly simple at present.
Here is the sample configuration file:
<?xml version="1.0"?>
<!-- configuration file for a feedbackd-master process -->
<feedbackd-master>
<!-- log configuration -->
<log>
<!-- where to log to -->
<file>/var/log/feedbackd-master.log</file>
<!-- log level. one of CRIT, ERR, WARN, INFO, DEBUG or VDEBUG -->
<level>WARN</level>
</log>
<!-- settings for the scheduler -->
<scheduler>
<!-- the time (in seconds) between periodic health checks ('keepalive'
messages) smaller values will give more up-to-date load information,
but will increase network load -->
<keepalive_interval>2</keepalive_interval>
<!-- how long (in seconds) before we consider a keepalive to be lost. After
sucessive (keepalive_retries) lost keepalive requests, the server is
considered dead and is quiesced -->
<keepalive_timeout>10</keepalive_timeout>
<!-- number of keepalive packets to retry before considering a server dead
-->
<keepalive_retries>3</keepalive_retries>
<!-- when a server is quiesced, it isn't removed from ipvs tables immediately,
as it may have active connections. feedbackd will check for active
connections every removal_interval seconds, and if the value is 0,
remove it. -->
<removal_interval>30</removal_interval>
</scheduler>
</feedbackd-master>
|
At present, the logging options are the only ones you need to worry about - the scheduler settings are covered in Section 6
Logging is configured by the <log> tag - which contains a tag for the verbosity level (the <level> tag) and the location to log to (one of <file>, <stdout> or <stderr>). The level parameter defines how verbose to be - it can be any one of the following levels, in order of increasing verbosity:
- CRIT
- Critial messages
- ERR
- Errors
- WARN
- Warnings
- INFO
- Informational messages
- DEBUG
- Debugging messages
- VDEBUG
- Verbose debugging
When first deploying feedbackd, log at DEBUG or INFO. Once you have it working, change to WARN to keep the size of the log files down.
When the master is informed that a realserver is available, it gets a list of the services running on that host. For each of these services, it looks for a matching virtual service in the director's ipvs tables. Therefore, you need to setup these virtual services before feedbackd-master is run, using ipvsadm -A. The following example will add a virtual service on the VIP for http (tcp port 80) requests:
director:# ipvsadm -A -t VIP:http -s rr |
Now you can run the master process. If you have compiled with ipvs support (the default behaviour, without the --without-ipvs option to ./configure), you will need to be root to run it.
director:# feedbackd-master |
feedbackd-master will parse the configuration, and if all is ok, run in the background. You can view the logs by:
director:# tail -f /var/log/feedbackd-master.log |
To stop the process, issue a:
director:# killall feedbackd-master |
| Note |
|---|---|
| The default behaviour on exit is to leave the ipvs tables as they are - the realservers will be weighted with the last value that was set. In the future (if there is any demand to do so), signals may be used to define the behaviour on exit (adjust all weights to be equal?) | |
Most of the options for feedback-master are defined in the configuration file. However, there are a few command-line options:
- -c FILE, --conffile=FILE
- Source configuration data from FILE instead of the default.
- -n, --dry-run
- Don't alter the ipvs tables (realservers are not added to ipvs tables, weights are not updated). The ipvs tables are still read for active connection count.
- -t, --test-config
- Test configuration data, then exit. This is useful after the configuration file has been altered
- -d, --dont-fork
- Do not run in the background - useful when logging to stdout.
- -V, --version
- Print version and exit
- -h, --help, -u, --usage
- Print usage information and exit
An agent process is installed on each realserver in the cluster to monitor realserver health and report it back to the master. The metric used to define health is configurable - a plugin is used to perform the actual measurement. The agents are also responsible for informing the master which services are available on each host.
An agent is required on each realserver. To start, you can setup and test the agent on one realserver, then copy the files to the other realservers.
Like the master, the agent requires the libxml2 library to parse configuration data. Also, perl is required for the perl.so plugin
The agent is installed in a similar way to the master:
realserver:$ tar zxvf feedbackd-agent-0.3.1.tar.gz realserver:$ cd feedbackd-agent-0.3.1 realserver:$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var realserver:$ make realserver:$ su realserver:# make install realserver:# $EDITOR /etc/feedbackd-agent.xml realserver:# feedbackd-agent -t realserver:# feedbackd-agent |
The feedbackd-agent source distribution is available from http://www.redfishsoftware.com.au/projects/feedbackd/. Untar the files into a build directory:
realserver:$ tar zxvf feedbackd-agent-0.3.1.tar.gz realserver:$ cd feedbackd-agent-0.3.1 |
Agent configuration is slightly simpler for the agent:
realserver:$ ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var |
The default settings for configure should be fine - if necessary, you can define the paths to the perl and xml-config programs:
- --with-xml-config=PATH
- Where to find the xml-config or xml2-config program. This is required to find the appropriate compiler flags and libraries to use for XML parsing.
- --with-perl=PATH
- Where to find the perl executable - required to configure the perl plugin
Building is the same as the master:
realserver:$ make |
Again, if this fails, send me an email with the output of make and your config.* files.
Building is the same as the master:
realserver:$ make install |
This will install the following files:
- The feedbackd-agent executable
- The configuration file, feedbackd-agent.xml
- The man page
- The monitor plugins and support files (by default these will be in /usr/lib/feedbackd-agent)
| Note | |
|---|---|---|
The --prefix argument (given to ./configure) is used in various stages of building, so changing the prefix in the install stage will not work. In short, do not specify an different prefix in the install stage:
| ||
The agent configuration file is installed in /etc/feedbackd-agent.xml by default. You'll need to change (at least) the director IP address, and perhaps the service definitions.
Here is the sample configuration file:
<?xml version="1.0"?>
<!-- configuration file for a feedbackd-agent process -->
<feedbackd-agent>
<!-- log configuration -->
<log>
<!-- where to log to -->
<file>/var/log/feedbackd-agent.log</file>
<!-- log level. one of CRIT, ERR, WARN, INFO, DEBUG or VDEBUG -->
<level>WARN</level>
</log>
<!-- how many connection attempts (to the master) to make before giving up.
0 = never give up (default) -->
<connection-retries>0</connection-retries>
<!-- where to find monitor plugins -->
<moduledir>/lib/feedbackd-agent/</moduledir>
<!-- where to connect to the master process. Either a hostname or an IP -->
<director>$DIP</director>
<!-- the defintion of a service. the name attribute is symbolic -->
<service name="http">
<!-- IP subprotocol to use, eg tcp or udp -->
<protocol>tcp</protocol>
<!-- port the service runs on. numeric ('80') or symbolic ('http'). -->
<port>http</port>
<!-- forwarding type, either DR, TUN or NAT -->
<forwarding>NAT</forwarding>
<!-- plugin to use for load monitoring. The contents of this
tag are passed to the monitor plugin to handle. -->
<monitor plugin="cpuload.so">
</monitor>
</service>
</feedbackd-agent>
|
Logging is configured in exactly the same fashion as the master - see Section 3.5.1.
The <director> tag gives the hostname or IP address of the director, where the master process is running. You will need to change this to suit your network.
The optional <moduledir> tag sets the path to find monitor plugin modules. This defaults to /usr/lib/feedback-agent if the tag is not present (with the default libdir given to configure).
The agent must inform the master of which services are available on this host, and each service needs a definition in the config file. There may be multiple service definitions, each with a <service> tag, with a symbolic name for the service given in the 'name' attribute.
Each service definition contains the following data:
- <protocol>
- The IP sub-protocol that the service uses; either TCP or UDP
- <port>
- The port that the service uses either in symbolic ('http') or numeric format ('80')
- <forwarding>
- The method used by the director to forward packets to the host. One of NAT, TUN or DR. (See the LVS documentation for details on forwarding types)
- <monitor>
- The monitor plugin to use to measure realserver health for this service. The monitor tag requires a plugin attribute, which defines the plugin to load. The contents of this tag depend on the exact plugin in use. See Section 4.7 for details on the available plugins
The example service given in the sample configuration file is for a http service (TCP port 80), with NAT forwarding. The cpuload plugin is used to measure realserver health - this plugin requires no extra configuration tags, so the monitor tag is empty.
Now you can run the agent process. Depending on the plugins and logfile in use, you may need to be root to run the agent.
realserver:# feedbackd-agent |
feedbackd-agent will parse the configuration, and if all is ok, run in the background. You can view the logs by:
realserver:# tail -f /var/log/feedbackd-agent.log |
After the agent has started, you should see a new realserver added to the ipvs tables on the director. To view these on the director:
director:$ ipvsadm -L |
Depending on the plugin in use, the weights will be updated on a regular basis
To stop the agent, issue a:
realserver:# killall feedbackd-agent |
| Note |
|---|---|
| When an agent exits, it informs the master that is it shutting down by sending a NECP_STOP packet. The master will then quiesce the realserver in the ipvs tables, so no new connections will be made to that realserver. Be careful when shutting down agents on all nodes of a live cluster - you may end up with no active realservers in the ipvs tables. | |
The feedbackd-agent has the same command line options as the master - see Section 3.6.1 for details.
Because the definition of "realserver health" changes between applications, feedbackd-agent uses dynamically-loaded plugins to measure the health of a service. Plugins can be added without requiring the agent to be rebuilt, and allow development of monitor plugins for custom applications. At present, the following plugins are available:
The constant module is the simplest plugin - it returns a (configurable) constant value for every measurement, defined in a <value> tag. For example:
<monitor plugin="constant.so"> <value>32</value> </monitor> |
The cpuload plugin reports the proportion of time that the CPU has spent idle. The higher this value, the 'healthier' the realserver is considered to be. This plugin requires no additional configuration tags:
<monitor plugin="cpuload.so"> </monitor> |
The perl plugin allows a monitor plugin to be implemented in a Perl script. The script to use is defined in a <file> tag:
<monitor plugin="perl.so"> <file>/usr/lib/feedbackd-agent/monitor.pl</file> </monitor> |
The Perl script must contain a get_health() sub, which returns a scalar between 0 and 100, representing the health of the service:
sub get_health() {
# perform necessary health measurement
return $health;
}
|
The Perl script is loaded and interpreted on startup, and the function is called when necessary. Therefore, altering the Perl file while feedbackd-agent is running will not alter the monitor code - feedbackd-agent must be restarted to reload the Perl file
Contributions for new plugins are welcome - either as a suggestion for development or a full implementation.
Because the feedbackd-master accepts connections from all remote hosts, it is possible for a malicious agent to connect to the master and add a realserver entry to the ipvs tables. This would allow a denial-of-service or man-in-the-middle attack on the services provided by the cluster. Therefore, it is important to block connections from unknown hosts to the director on tcp port 3262.
There are a number of parameters that allow optimisation of feedbackd behaviour to suit specific applications and network topologies.
The frequency of the NECP keepalive messages can be altered on the master. Lower values of the interval will give the director more up-to-date load data, but will increase network and processor load.
The keepalive interval can be changed in the <keepalive-interval> tag of the feedbackd-master configuration file. The interval is given in seconds.
| Note |
|---|---|
| The keepalive messages also inform the agent that it still has an active connection to the master. See Section 6.3. If keepalive-interval is higher than master-timeout, then the agent will presume that the connection has been broken and will disconnect from the master and attempt a reconnection. | |
The default value for the keepalive interval is 2 seconds
If a response is not received for a keepalive message within a predefined amount of time, then the keepalive message is considered lost and resent. After a predefined number of successive retries, the master assumes that the remote host is down, and will quiesce the realserver (so that future requests are not forwarded to the realserver).
These two predefined values are the "keepalive timeout" and "keepalive retries", respectively. These are found in the master configuration file, in the <keepalive-timeout> (value given in seconds) and <keepalive-retries> tags.
These values should be adjusted to suit the network connection between the director and realservers. If the realservers are only reachable through high-latency links, then the timeout value should be higher, to prevent active realservers from being inadvertantly quiesced. A timeout that is too high, however, will result in faulty realservers being quiesced after a longer amount of time (giving a larger proportion of errors in the load balanced service).
The default value for the keepalive timeout is 10 seconds, the default number of retries is 3.
(More specifcally, Master Timeout as seen by the agent on the realserver)
The agent expects to receive regular keepalive messages from the master. If the agent receives no data from the master before the timeout, then the agent will immediately close the current connection to the master and reconnect. If you are logging at a level at or above WARN, then the following message will be logged:
Master timed out, attempting reconnection |
If these messages occur frequently, you may have the master timeout set too low - the agent is timing out before the master has had a chance to send the keepalive packet. However, large values will result in a slow detection of broken connections.
The master timeout is set in the feedbackd-agent config file, in the <master-timeout> tag. The value is given in seconds.
| Note |
|---|---|
| If the master timeout is lower than (or close to) the keepalive interval, then the agent may timeout unnecessarily. Make sure the master timeout value is not too low. | |
The default value of the master timeout is 30 seconds