Circonus Enterprise Brokers

Overview

Circonus operates a worldwide system of nodes for configuring checks and gathering metrics. These nodes are known as Circonus Public Brokers and are available for use by any active Circonus accounts. Enterprise Brokers are software appliances that can be downloaded and installed in a customer's datacenter for running checks and gathering metrics privately.

System Requirements

  • RHEL/CentOS 6/7, OmniOS r151014 LTS
  • 2 CPU cores
  • 4 Gbytes of RAM
  • 40 Gbytes of disk storage

Installation

On all platforms, the necessary services will be started automatically upon package installation.

RHEL 6 / CentOS 6 Installation

Follow these instructions to install an Enterprise Broker on RHEL 6 or CentOS 6.

Note:

Only 64-bit (x86_64) is supported for RHEL/CentOS 6

Create a file at /etc/yum.repos.d/Circonus.repo with the following contents:

# Circonus.repo 
#
[circonus]
name=Circonus - Base
baseurl=http://updates.circonus.net/centos/6/x86_64/
enabled=1
gpgcheck=0
metadata_expire=60m

[circonus-crashreporting]
name=Circonus - Crash Reporting
baseurl=http://updates.circonus.net/backtrace/centos/el6/
enabled=1
gpgcheck=0
metadata_expire=60m

There is a single package to install via YUM which will pull in others as dependencies. Use the following command:

# yum install circonus-field-broker

RHEL 7 / CentOS 7 Installation

Follow these instructions to install an Enterprise Broker on RHEL 7 or CentOS 7.

Create a file at /etc/yum.repos.d/Circonus.repo with the following contents:

# Circonus.repo 
#
[circonus]
name=Circonus - Base
baseurl=http://updates.circonus.net/centos/7/x86_64/
enabled=1
gpgcheck=0
metadata_expire=60m

[circonus-crashreporting]
name=Circonus - Crash Reporting
baseurl=http://updates.circonus.net/backtrace/centos/el7/
enabled=1
gpgcheck=0
metadata_expire=60m

There is a single package to install via YUM which will pull in others as dependencies. Use the following command:

# yum install circonus-field-broker

OmniOS Installation

Follow these instructions to install an Enterprise Broker on OmniOS.

Note:

The r151014 (LTS) version is required.

First, add the Circonus IPS publisher using the following commands:

# pkg set-publisher -g http://updates.circonus.net/omnios/r151014/ circonus
# pkg set-publisher -g http://updates.circonus.net/backtrace/omnios/ backtrace

Next, install the Broker package using the following command:

# pkg install field/broker

External Connectivity

In order to securely communicate with the Circonus service and publish metrics, the Enterprise Broker requires a secure SSL connection with the Circonus metric aggregation system. If the appliance resides behind a firewall, you should add a rule to your firewall allowing Circonus IP addresses to reach the Enterprise Broker over TCP port 43191. The list of IP addresses from which Circonus traffic originates may be obtained via DNS lookup of out.circonus.net. You should periodically check the result of this DNS lookup and validate your firewall rules, as the IP addresses are subject to change.

It is also possible to configure the broker to initiate a connection to Circonus, using the -nat option (see below.)

The Enterprise Broker should be allowed to respond to these connections. No other outbound connectivity initiated by the Broker is required, although it may be desired for running outbound checks from the Broker.

Provision the Broker

To provision a Broker once the Broker is installed, you will use a command line tool called "provtool". Once it is provisioned, you should be able to deploy new checks onto the Enterprise Broker using the Circonus check wizard or Management API.

Note:

If you are using Circonus Inside, then you must set the api-url before running the provisioning tool. Use the following command:

sudo /opt/napp/bin/provtool config set api-url https://api.your.inside.install

This is the general purpose provisioning process:

  1. Obtain an API token that has Admin privilege.
  2. Stop the noitd service
  3. sudo /opt/napp/bin/provtool config set api-token <ADMIN_USER_API_TOKEN>
  4. sudo /opt/napp/bin/provtool provision -ip <publicip>
  5. Start the noitd service

To provision a Broker using the Provtool, run the command:

provtool provision [-cn <cn>] [-ip <ip>] [-name <name>] [-nat]

where <cn> is the broker CN ("Common Name", for example, a000-n123456.noit.circonus.net), <ip> is the broker IP address to which Circonus will connect, and <name> is the name of the broker. Naming the broker is optional. The -nat option will trigger the broker to connect to Circonus instead of Circonus connecting to the broker. Other Provtool Options are detailed below:

Provtool Options:

  • Local configuration

    provtool config get <key>
    provtool config set <key> <value>
     api-token:    the Circonus API token for provisioning
                 (token must have Admin privilege)
     api-url:    the Circonus API base url
                 (default: https://api.circonus.com)
     googleanalytics/api-key:    Google Analytics API Key
     googleanalytics/client-id:    Google Analytics Client Id
     googleanalytics/client-secret:    Google Analytics Client Secret
    
  • Listing brokers

    provtool list
    
  • Provision this broker

    provtool provision [-cn <cn>] [-ip <ip>] [-name <name>]
     -cn <cn>    specify a broker CN, default first unprovisioned
     -ip <IP>    set the broker IP address to which Circonus will connect
     -long <longitude>    set the broker's longitude
     -lat <latitude>    set the broker's latitude
     -name <name>    an optional name for the broker
     -nat        tell Circonus that this broker will dial in
    
  • Rebuilding a broker's configuration

    provtool rebuild [-c <cn>]
     -c <cn>    rebuild an arbitrary cn [deault: this machine].
    

Specifying a broker slot

When you provision a new broker, the provtool will automatically find and use an unprovisioned broker slot. If you want to specify a broker slot, you can review the list of broker slots with a command, sudo /opt/napp/bin/provtool list, and then use the CN for the broker slot you desire via the -cn option when you provision the broker. However, note that this can be dangerous, because if you specify a CN that is already in use, then the broker will rekey that slot so you can "rebuild."

Rebuilding a failed broker

This is the method for rebuilding a failed broker.

  1. Stop the noitd service
  2. sudo /opt/napp/bin/provtool config set api-token <AUTH_TOKEN>
  3. sudo /opt/napp/bin/provtool provision
  4. Start the noitd service

If the IP address cannot be detected, -ip <publicip> is required in the provision step.

If Circonus cannot connect to the broker and the broker should instead connect to Circonus specify the -nat option. The -name option is not required, but can be used to name the broker. The -long and -lat options are not required, but can be used to set the broker's global location.

If the broker has already been activated and has a configuration, but the box is a fresh broker intended to be used to recover for a failed broker, then follow these steps:

  1. Specify -cn <broker to rebuild> on the provision line. The broker to rebuild should be an active broker in the system that is now permanently offline. This new broker will assume all of the old broker's responsibilities.
  2. Stop the noitd service
  3. sudo /opt/napp/bin/provtool config set api-token <AUTH_TOKEN>
  4. sudo /opt/napp/bin/provtool provision -cn <broker to rebuild> -ip <publicip> ....
  5. sudo /opt/napp/bin/provtool rebuild
  6. Start the noitd service

Updating

Package updates from Circonus are periodically available for Enterprise Brokers.

When an Enterprise Broker receives an "Update Software" message, use one of the following commands to install the update, depending on the Broker's operating system:

  • RHEL/CentOS 6/7:
    yum update circonus-field-broker
    
  • OmniOS:
    pkg update field/broker
    

Reinstallation

If it becomes necessary to reinstall the Broker on a new machine, having the existing Broker available makes the process simple, but Circonus support can still help you restore your checks even if you have completely lost a Broker system. (Contact support@circonus.com.)

Warning:

Do NOT decommission the current broker while performing a reinstallation under any circumstances, unless instructed to do so by Circonus support.

Note:

Your Broker status page may show a software out-of-date message when you initially start up the new Broker. This can take up to 15 minutes to clear.

Current Broker Available

Follow these instructions for reinstallation when the current Broker is available:

  1. Install the new Broker using the installation instructions above.
  2. Copy the contents of /opt/napp/etc/ssl to the new machine.
  3. Make sure the noitd process is stopped on the existing broker, we will not be turning it back on.
  4. Copy the contents of /opt/noit/prod/etc/ to the new machine. At this point, the new broker is ready to boot up and start collecting data. The next steps will disconnect your existing broker from Circonus and connect the new one.
  5. Navigate to your account's Broker status page.
  6. Click on the Broker in question and update its IP to the new machine. Note that both the old and new Brokers should be running at this point, when you enter the new IP, Circonus will reach out to the new Broker to make sure it can talk to it. If it can not, you will receive an error message stating that the system could not update the Broker at this time.
  7. Now you may stop the noitd process on the current Broker.

You should see your new Broker connected on the status page. If you have any problems, please contact Circonus Support (support@circonus.com).

Current Broker Not Available

If the current broker is no longer available, use the Provtool (/opt/napp/bin/provtool) and follow the instructions for "Rebuilding a failed broker" above.

Services

The broker package provides two services: "noitd" and "jezebel". Both are enabled automatically during package installation.

noitd

This is the primary monitoring daemon. It runs as a supervisor process with one or more child processes that actually perform checks. If a child process crashes, the supervisor will start another one, but if too many crashes happen in too short a time, the supervisor will stop itself rather than continue an endless cycle of restarts.

To start, stop, or restart:

  • RHEL/CentOS 6: /sbin/service noitd {start|stop|restart}
  • RHEL/CentOS 7: /usr/bin/systemctl {start|stop|restart} noitd
  • OmniOS: /usr/sbin/svcadm {enable|disable|restart} noitd

To check status:

  • RHEL/CentOS 6: /sbin/service noitd status
  • RHEL/CentOS 7: /usr/bin/systemctl status noitd
  • OmniOS: /usr/bin/svcs -p noitd

jezebel

This is a helper process, written in Java. It performs certain types of checks, mostly database checks. The noitd process communicates with jezebel over a local TCP connection, directing it to perform checks and collecting the metric data that is returned.

To start, stop, or restart:

  • RHEL/CentOS 6: /sbin/service jezebel {start|stop|restart}
  • RHEL/CentOS 7: /usr/bin/systemctl {start|stop|restart} jezebel
  • OmniOS: /usr/sbin/svcadm {enable|disable|restart} jezebel

To check status:

  • RHEL/CentOS 6: /sbin/service jezebel status
  • RHEL/CentOS 7: /usr/bin/systemctl status jezebel
  • OmniOS: /usr/bin/svcs -p jezebel

Important Files and Directories

  • /opt/napp/etc/ssl This is the location of SSL key and certificates, including the broker's client certificate. It is used for communicating with the Circonus infrastructure and as a CA certificate for authenticating connections from Circonus. All files in this directory should be backed up.

  • /opt/noit/prod/etc This location is for configuration files. In general, you should not need to manually edit any of these file, with a couple of exceptions, noted below. Changes to editable files will be preserved during broker package updates.

    • circonus-modules-enterprise.conf may be edited to configure/enable/disable enterprise-related check modules such as collectd, statsd, and cloudwatch.
    • circonus-modules-site.conf may be updated to activate custom noitd modules that you create. See the section below entitled "Configuring a Custom Module with Reconnoiter" for details about custom modules.
  • /opt/noit/prod/etc/(checks,filtersets) These directories contain the individual check configurations assigned to this broker. They are created, updated, and removed automatically by noitd and should not be changed manually.

  • /opt/noit/prod/log This location contains log files.

  • /opt/noit/prod/log/noitd.feed Journaled log of collected metric data.

Logs

Logs are written under /opt/noit/prod/log. There are two types of log files: access.log and noitd.log.

Access logs record operations on the broker's check configurations, as well as push operations such as HTTPTrap. The noitd log records activity about the noitd process itself, including startup messages and information about child processes that crash. These files are rotated automatically when they reach approximately 10MB in size and a total of about 5 historical files for each log will be retained.

Metric Feed Log

Also in the same directory is noitd.feed. This subdirectory contains the journal of collected metric data that will be sent to Circonus. It is implemented with JLog, which allows multiple "subscribers" (Circonus metric aggregators) to read metric data, maintaining an individual checkpoint for each subscriber. If connectivity to the broker from Circonus is lost, metric data will accumulate in the feed directory until connectivity is restored.

If the contents of this directory are lost before they are consumed by Circonus, data for affected metrics will be permanently lost. Care should be taken to ensure sufficient disk space for this directory to grow in the event of a loss of connectivity. Disk space requirements grow as the number of checks and metrics configured on the broker increases.

Configuring a Custom Module with Reconnoiter

Note that these instructions may vary for Linux and OmniOS.

All Broker (Reconnoiter) configuration information is stored on the Broker machine in the path /opt/noit/prod/etc. Users can load custom modules by modifying a file in this path named circonus-modules-site.conf. This .conf file will not be overwritten by broker updates.

To add a custom module, simply add the name of the script or loadable object to the site modules config file, then restart the "noitd" service. Configuration changes will take effect when noitd is restarted.

For example, loading a lua module called "myluamodule" and a C module called "mycmodule" would look like this:

<?xml version="1.0" encoding="utf8"?>
<lua loader="lua">
<module name="myluamodule" object="noit.module.myluamodule"/>
</lua>
<module name="mycmodule" object="mycmodule"/>

This lua module would be delivered in a file called "/opt/noit/prod/libexec/noit/lua/noit/module/myluamodule.lua"

This C module would be delivered in a file called "/opt/noit/prod/libexec/noit/mycmodule.so".

For information about how to develop modules, refer to the Reconnoiter Manual or view it externally on the OmniTI Labs site.

Troubleshooting

Check noitd Status

Access the broker machine and use following the commands:

telnet localhost 32322

followed by

show checks

This will display all the checks configured on the broker. To inspect an individual check, you use the command

show check <uuid>

This will display information such as when the check last ran and when it is due to run again. This will help determine if things are running properly.

Activating Automated Crash Reporting

If you would like to help Circonus improve the quality of the broker software, you may activate an automated reporting process to send details of crashes to Circonus for analysis. This requires that the broker machine be able to connect out to https://circonus.sp.backtrace.io:6098 to send reports.

To activate automated crash reporting on RHEL/CentOS 6/7:

yum install circonus-field-broker-crashreporter

To activate automated crash reporting on OmniOS:

pkg install field/broker/crashreporter

A new service called "circonus-coroner" (RHEL/CentOS) or "svc:/circonus/coroner" (OmniOS) will be installed, which will watch for noitd crash events and report them. Each time a crash is noted, a summary of the trace will be written to a file in /opt/noit/prod/traces with a ".trc" extension. The .trc files are a local record of trace events for informational purposes.

There will also be, briefly, a report file with a .btt extension, which is what coroner will upload to Circonus, and then remove. If you see .btt files lingering in the traces directory, check that the coroner service is running, and that you have the necessary network connectivity available.

Time Synchronization problems with VirtualBox

Under load, VirtualBox on Linux hosts can cause the system clock to temporarily hang and upset the VM guest. This can cause major problems with the Broker process (noitd). A possible solution is to disable host/guest time syncing.

First, install the VBox Guest Additions for Linux in the Enterprise Broker (see http://forums.virtualbox.org/viewtopic.php?t=4960) and shut down the appliance.

Next, run the following command to disable host/guest time syncing:

"VBoxInternal/Devices/VMMDev/0/Config/GetHostTimeDisabled" "1"

Finally, boot the Enterprise Broker.

Resetting the Agent to Factory Defaults

Once a Broker is provisioned and in use, the way to "start over" with a fresh Broker is to decommission the current Broker via the Circonus UI and create a new one.

Warning:

Decommissioning a Broker deletes all checks associated with the broker, along with all other traces of it in the Circonus system. If you simply wish to relocate a Broker to another machine, please see the Reinstallation section above.

To decommission a Broker, open the main menu and navigate to "Account: Brokers", then open the details pane for the Broker in question. Hover over the "bomb" icon in the lower right corner, and the "Decommission Broker" button will appear. Click this button to initiate the decommissioning process.

If you wish to reuse the same machine that you are currently using, you must remove all Circonus packages and delete the /opt/napp and /opt/noit directories after you have decommissioned it.

Sending Files to Circonus Support

When contacting Circonus Support (support@circonus.com) for assistance with Broker troubleshooting, you may be asked to upload a file directly to Circonus's servers.

The instructions for this procedure can be found in the Tech Support Appendix.

results matching ""

    No results matching ""