libut Primer
Troy Hanson
This document is a draft. Portions are under development.
Background
Purpose
Libut is a C library for writing UNIX servers, or daemon processes.
Build and Install
Once the libut distribution has been downloaded and untarred, cd into the top-level directory and run
./configure make make install
You can run configure --help for additional build and install options. The make install step is not necessary if you wish to use libut from the directory in which you untarred it.
Simplest server: idle
This is a fully functional server with built-in logging, runtime configurability and an administrative control port:
#include "libut/ut.h"
int main() {
UT_init(INIT_END);
UT_main();
}
Try it: run the idle server
If you did the Build and Install, you can run idle in the tests/idle directory of the libut distribution. You should see log messages appear as the server is running. Type Control-C to exit.
Platform
libut is written in C, compiles as a static library, and builds on Linux, Solaris, OpenBSD, and Mac OSX.
What kind of servers can be developed using libut?
Libut provides generic logging, configuration and administration facilities, as well as API support for TCP socket communication (server or client), timers, hashtables, and more. It is meant to be usable for typical event-driven UNIX daemon development. It can also be subservient to another framework which owns the main loop and calls libut periodically.
Design
Non-threaded, non-blocking, event-driven design
select()-based non-blocking I/O
Libut uses the Unix select() call to manage I/O on an arbitrary number of file descriptors simultaneously. There is no need for threads. When I/O availability on any file descriptor(s) is indicated, libut invokes read/write callbacks for each descriptor. This I/O is conducted in non-blocking mode. This means that a callback does not block in a read() waiting for unarrived data, or block in a write waiting for the reader at the "other end" to catch up. Reads and writes take place as the descriptor becomes ready.
No threads
It bears repeating that there are no threads in libut. It is not necessary to thread a program to handle multiple open I/O connections because select() alerts our server to available I/O on any of a whole set of file descriptors simultaneously; and non-blocking I/O relies on the kernel (which may be threaded) to complete background writes and to buffer pending reads. Since threads are not used, no locking of data structures or code sections is necessary when writing a libut-based server.
Timers
Timers are the other way (besides I/O callbacks) that libut-based servers do their work. Timers are simple callbacks which are invoked a specific number of milleseconds after they're created. Timers can be repeating.
Event-driven
Libut is event-driven meaning that its main loop recognizes events (file descriptor I/O availability or timer expirations) and invokes callbacks to handle those events. Event-driven programs are asynchronous; their I/O is done piecemeal and there is no synchronous requirement for I/O on one descriptor to finish before I/O on another descriptor can take place. As a result, almost all libut-based server functionality is implemented in the form of callbacks for various types of events.
The Control Port
Status and configuration commands can be issued to a running libut-based server.
% telnet localhost 4445 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. libut configuration interpreter (shl) Type 'help' for command list. help command description ----------------- ------------------------------------- * mem - memory pool usage summary * var - Display or set config variables * log - Change log file or verbosity fds - list selected file descriptors tmr - show pending timers uptime - show uptime * prf - Performance/profiling stats * cops - List coprocesses help - view command help exit - close connection Commands preceded by * have detailed help. Use help <command>. Ok
The control port gives visibility into the server's inner state
Feel free to experiment with the control port commands. They are there to give the administrator and developer visibility into the inner state of the server. The developer can further enhance this visibility by defining additional commands suitable to the particular server.
For administration, development and testing
Libut emphasizes the use of the control port as a unified interface for run-time administration/configuration, but also for development/testing. Commands can be defined to test features as they're being developed.
Connecting to the control port
Connecting to the control port is easy:
telnet localhost 4445
This connects to the control port's default address and port. 1
Try it: connect to the idle server's control port
If you did the Build and Install, run the idle server located in the tests/idle directory. Then, in another terminal window, telnet localhost 4445 to connect to the control port. Type help to see the built-in commands. Try them out, then type exit to disconnect. Type Control-C in the original window to terminate the server.
ABC's of libut
Initialization
UT_init() must be called prior to any other libut API function. This function takes arguments that specify initialization options.
server.conf configuration file
Libut looks for a server.conf configuration file (this default name is configurable via a UT_init() option) in the current directory at startup. If found, it is evaluated. The same commands and syntax that are used in the control port are accepted in the configuration file.
Configuration variables
Libut-based servers have configuration variables. A number of configuration variables are pre-defined. Developers add their own using the UT_var_create() API function.
The var command
The var control port command displays the values of all configuration variables.
var name description value --------------------- ------------------------------ -------------------- *ut_log_level log level Debugk *ut_log_file log file /dev/stdout *ut_jobname job name job1 *ut_control_port control port IP/port 127.0.0.1:4445 *ut_basedir shl base directory /home/thanson/code/libut2 Variables prefixed with '*' can be changed. Ok
Used with extra parameters, it also sets them. E.g.,
var ut_log_file = /tmp/log
Accessing configuration variables from C server code
Server code can read, watch or set configuration variables using API functions UT_var_get(), UT_var_reg_cb(), UT_var_bind_cvar(), and UT_var_set(). Additionally the server can restrict the values for a variable using UT_var_restrict().
Logging
Libut-based servers use the UT_LOG macro to write messages to a log. The log file has lines of this format:
Aug 26 00:38:51 Info (listen.c:224) control_port listening on 127.0.0.1:4445
Log file location
The ut_log_file configuration variable specifies where to write the log. By default the log is written to standard output. This default can be changed by passing appropriate options to the UT_init() API function. The default can be overridden by setting the configuration variable in the control port or in the server.conf configuration file.
Log levels
Each log message is logged at one of these five levels: Messages at the current log level and those more severe are logged. The ut_log_level configuration variable specifies the current log level.
Fatal log level
Writing a log message at the Fatal log level causes the server to exit after writing the log message. This is a convenient way to handle non-recoverable situations without having to code the exit() manually.
log control port command
In the control port, you can "bump" the log level up or down. E.g.,
log more log less
Memory pools
At initialization, libut-based servers define memory pools for each of the data structures they routinely need to allocate and free. A pool is defined using the UT_mem_create_pool() API function.
Allocation and freeing of buffers
The server can allocate and free buffers (i.e., structures) from a pool using the UT_mem_alloc_buf() and UT_mem_free_buf() API functions. The advantage of using these calls instead of malloc() and free() is monitorability: the mem control port command displays buffer usage (and kilobytes consumed) for all pools and for the server as a whole.
mem pool bufs_used bufs_total kb_used kb_total --------------------------------------------------------------------------- ut_hash_tbl 5 10 0.2 0.3 ut_var 5 10 0.7 1.4 ut_callback 5 10 0.1 0.1 ut_fd 3 10 0.4 1.2 ut_tmr 0 10 0.0 0.4 ut_prf 1 10 0.2 1.7 ut_prf_row 1 10 0.1 0.9 ut_prf_bkt_cnt 6 100 0.0 0.4 ut_prf_bkt_top 6 10 0.0 0.1 ut_coprocess 0 10 0.0 0.5 ut_shl_session 1 1 0.0 0.0 ut_shl_cmd 10 10 1.1 1.1 ut_iob 18 100 0.2 1.2 ut_request 0 10 0.0 0.2 ut_per_loop_cb 0 2 0.0 0.0 Total(KB) 2.9 9.4 Ok
Out-of-memory handling
If memory cannot be allocated, libut logs a Fatal message and exits. There is no need for the server code to check if an allocation succeeded.
Organization of a memory pool
Internally, each memory pool consists of a set of rows. Each row has a number of buffers. Memory is allocated to the pool in whole rows at a time. Most rows have a standard number of buffers in each row (a parameter passed in when the pool is defined). But an allocation request for more buffers than are present in a standard row creates a row with extra buffers.
The mem poolname control port command displays these characteristics.
mem ut_iob This pool has 2 buffer rows Each row has 10 buffers (0 exceptions) Each buffer is 12 bytes Of 20 total buffers (occupying 0.2 kb), 3 are in use In this pool's lifetime there have been 34 buffers requests, 28 frees Ok
Profiling
The libut profiling functions allow the developer to surround any section of code with a pair of API calls to monitor the ongoing execution time of that code. The prf control port command displays these profiles.
prf
=================================================================
database -- probase query timings
=================================================================
name count <1m 1-10m 10-100m 0.1-1s 1-10s >10s
--------------------- ------- --- ----- ------- ------ ----- ----
database-read 10 60% 40%
database-write 2 30% 70%
Ok
In this example, the server (at startup) defined the "database" group using UT_prf_create(), and the database read and write code have each been surrounded by a pair of calls to UT_prf_bgn() and UT_prf_end() to delimit the timing interval.
Time scales
Libut supports several built-in time scales. There are linear, logarithmic and non-uniform time scales. The time scale is an argument to UT_prf_create().
The event loop
UT_event_loop() entry point
The UT_event_loop() API function enters the event loop. Once entered, the event loop never returns. Each iteration of the loop consists of invoking callbacks for any available I/O or expired timers.
Alternative: Embedding the libut event loop
There is an alternative to UT_event_loop() called UT_event_loop_once() which executes one iteration of the event loop: UT_event_loop_once(). This call can be used to embed libut's event loop within an external main loop. The external main loop only needs to call UT_event_loop_once() periodically. 2
Event loop operation sequence
The event loop consists of three parts.
Per-spin callbacks
First per-spin callbacks are executed. These are rarely used.
File descriptor I/O availibility
Second, I/O callbacks are invoked for any file descriptors being monitored for I/O readiness.
Timers
Third, timer callbacks are invoked for all expired timers.
Linked lists
Libut has a set of macros for working with linked lists of structures. Any type of structure can be used, but it must contain a "next" pointer. The macros permit structures to be added, deleted, and looked-up in a list.
The macros only work as standalone statements. I.e. they should not be used as a conditional for an if(...) statement, etc.
List head
The list head must be a pointer which has been initialized to NULL.
LL_ADD
Appends an element (a structure) to a linked list. All three arguments are pointers to the same type of structure.
LL_ADD(head,tmp,add); head - list head tmp - for internal use (need not be initialized) add - pointer to the structure to add to the list
LL_DEL
Deletes an element (a structure) from a linked list. All three arguments are pointers to the same type of structure.
LL_DEL(head,tmp,del); head - list head tmp - for internal use (need not be initialized) add - pointer to the structure to delete from the list
LL_FIND
Find an element (a structure) in a linked list by name. This macro requires that the structure have a (char*)name; field.
LL_FIND(head,tmp,name); head - list head tmp - afterward, this contains pointer to found element (or NULL) name - string to search for in ''name'' field of each list element
LL_FIND_BY_MEMBER
Find an element (a structure) in a linked list by a non-string field.
LL_FIND_BY_MEMBER(head,tmp,mbr,val); head - list head tmp - afterward, this contains pointer to found element (or NULL) mbr - structure field to search val - look for structure whose |mbr| field has this value (==)
LL_FIND_BY_MEMBER_STR
Find an element (a structure) in a linked list by a string field.
LL_FIND_BY_MEMBER_STR(head,tmp,mbr,str); head - list head tmp - afterward, this contains pointer to found element (or NULL) mbr - structure field to search str - look for structure whose |mbr| field has this value (strcmp)
Hashes
Hashes are easy to use and much more efficient than linked lists when searching a large number of structures for the one you want. In libut, hashes are structures of any type. One field (identified by its name when invoking the macros) is considered the key field. The key field can be integer, string or a binary n-length type. The key uniquely identifies the structure; no other structure in the hash can have the same key.
Hashes can be iterated upon as linked lists
Each hash is also a linked-list whose elements (structures) are in the order they were added by the application. Iterate over the structures by starting at the hash head and following the next pointer until NULL.
Hash head
The hash head must be a pointer which has been initialized to NULL.
Required fields
The structure must contain a UT_hash_handle hh; field. (Its name has to be hh as shown.) In addition the structure must contain a "next" pointer. Any fields other than these two and the key field are optional.
HASH_ADD
HASH_DEL
HASH_FIND
iob's
A UT_iob data structure is a dynamic memory buffer. It can contain arbitrary data (binary or string) of any length and be appended to. Internally it is implemented as a linked-list of buffers.
The name iob is short for input-output buffer, but they can be used anywhere dynamic buffers are needed, not just for input/output.
Server code can use UT_iob_printf() to write string data to a UT_iob() or place binary data into one using UT_iob_bincpy(). These calls can be used repeatedly and intermixed; each one appends data. See UT_iob_create, UT_iob_free, UT_iob_printf, UT_iob_bincpy, UT_iob_flatten, and UT_iob_len.
Coprocess support
In UNIX, when process A has a pipe to the standard input of process B, and the standard output and standard error of B are also piped back to A, process B is called a coprocess of process A.
For delegated work
Coprocesses can be useful when a libut-based server needs to "delegate" work that the server does not want to do (or can't do) itself. Typically the coprocess may do a blocking operation such as a database query.
Coprocesses are forked from the server
The API function UT_fork_coprocess() is used to fork a coprocess.
The coprocess executes a specified function, then exits
One parameter of UT_fork_coprocess() specifies the entry function to be called in the new coprocess. This function could then exec() to turn itself into another program. Otherwise, when the function returns, the coprocess exits. The return value from the entry function becomes the exit status.
The coprocess does not run the libut event loop
Coprocesses cannot return from their entry function (if they do, the coprocess exits), therefore the coprocess cannot return to the event loop. The intended use of a coprocess is to run "delegated" work that is generally of a blocking nature or to execute a separate binary.
The server can communicate with the coprocess
The server has three pipes to the coprocess:
- a pipe to write to its standard input
- a pipe to read from its standard output
- a pipe to read from its standard error.
The file descriptors for the first two pipes are output parameters of the UT_fork_coprocess() API function. Libut has special built-in handling for the third (standard error) pipe.
Libut automatically logs standard error from the coprocess
When the coprocess writes to its standard error, the lines that it writes are automatically logged in the server process at the Info level.
The idea is that a coprocess can write its errors or important messages to standard error and rely on them being logged in the server-- because the coprocess has no log file or console of its own.
In this example, errors from a coprocess named "dbquery" are being logged: 3
Aug 28 10:58:50 Info (coproc.c:100) dbquery: [database returned error 2] Aug 28 10:58:50 Info (coproc.c:100) dbquery: [closing data]* Aug 28 10:58:50 Info (coproc.c:100) dbquery: [base connection]
Coprocesses are automatically collected when they exit
When a coprocess finishes its work, it can either call exit() or return from its entry function. In either case the coprocess exits. The server is notified of the exit of the coprocess, and libut collects it automatically.
An exit callback can be specified
A callback to run when the coprocess exits can be specified as one of the arguments to UT_fork_coprocess(). The exit statusof the coprocess is made available to this callback.
cops lists the coprocesses
The cops (coprocess status) control port command lists information about each coprocess that is currently running.
API
Miscellany
Name
What does the ut in libut stand for? It once meant utility toolkit but that doesn't really fit anymore. Its more accurate to call it a server development library.
History
Troy Hanson developed libut starting in 2003.
LICENSE
Copyright (c) 2003-2005, Troy Hanson All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Example Control Port Transcript
This transcript was produced using the idle application included in the libut distribution in the tests/idle directory. If you compiled libut and its tests (by running configure; make in the top-level directory) you can run the tests/idle/idle executable and try this yourself.
% telnet localhost 4445
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
libut configuration interpreter (shl)
Type 'help' for command list.
help
command description
----------------- -------------------------------------
* mem - memory pool usage summary
* var - Display or set config variables
* log - Change log file or verbosity
fds - list selected file descriptors
tmr - show pending timers
uptime - show uptime
* prf - Performance/profiling stats
* cops - List coprocesses
help - view command help
exit - close connection
Commands preceded by * have detailed help. Use help <command>.
Ok
uptime
program started: Wed Aug 24 01:17:18 2005
current time: Wed Aug 24 01:18:32 2005
up 0 days, 0 hours, 1 min, 14 sec
Ok
var
name description value
--------------------- ------------------------------ --------------------
*ut_log_level log level Debugk
*ut_log_file log file /dev/stdout
*ut_jobname job name job1
*ut_control_port control port IP/port 127.0.0.1:4445
*ut_basedir shl base directory /home/thanson/code/libut2
Variables prefixed with '*' can be changed.
Ok
help var
Usage:
var -- show all vars
var varname -- show named var
var varname [=] newval -- set var to new value
Ok
prf
=================================================================
libut -- libut statistics
=================================================================
name count <1m 1-10m 10-100m 0.1-1s 1-10s >10s
--------------------- ------- --- ----- ------- ------ ----- ----
i/o callbacks 23 65% 30% 4%
Ok
fds
fd name type flg io-count md local/remote address
-- ---------------- ---- --- -------- -- ------------------------------------
4 control_port sock L 3 r 127.0.0.1:4445
5 control_port sock A 6 r 127.0.0.1:4445 127.0.0.1:1346
Ok
mem
pool bufs_used bufs_total kb_used kb_total
---------------------------------------------------------------------------
ut_hash_tbl 5 10 0.2 0.3
ut_var 5 10 0.7 1.4
ut_callback 5 10 0.1 0.1
ut_fd 3 10 0.4 1.2
ut_tmr 0 10 0.0 0.4
ut_prf 1 10 0.2 1.7
ut_prf_row 1 10 0.1 0.9
ut_prf_bkt_cnt 6 100 0.0 0.4
ut_prf_bkt_top 6 10 0.0 0.1
ut_coprocess 0 10 0.0 0.5
ut_shl_session 1 1 0.0 0.0
ut_shl_cmd 10 10 1.1 1.1
ut_iob 18 100 0.2 1.2
ut_request 0 10 0.0 0.2
ut_per_loop_cb 0 2 0.0 0.0
Total(KB) 2.9 9.4
Ok
exit
Connection closed by foreign host.
%
[1] - By default the control port listens on port 4445 of the loopback address 127.0.0.1. (The loopback address usually has the name localhost). When a socket listens on the loopback address, it is not listening on the network. Therefore, only a process on the same host can connect to it.
[2] - it should be called frequently enough to service the I/O and timer requirements of the server, otherwise the server may become sluggish.
[3] - If a partial line (lacking a trailing newline) is read from the coprocess, the server logs it with a trailing asterisk (*). Even though the coprocess may have written a full line, it may be read by the server in more than one fragment.
This document was generated using AFT v5.095