forked from boyska/circolog
109 lines
5 KiB
Markdown
109 lines
5 KiB
Markdown
A syslog daemon implementing circular buffer, in-memory storage.
|
|
|
|
This is useful when you want to keep some (heavily detailed) log available, but you don't want to log too many
|
|
things to disk. Remember: logging is useful, but can be dangerous to your users' privacy!
|
|
|
|
On your "main" syslog, forward (part of the) messages to this one!
|
|
|
|
## Integration examples
|
|
|
|
In these examples I'll refer to the usage of UNIX sockets. They are more secure than TCP/UDP sockets because
|
|
they have file permissions, they can be "masked" using mount namespaces, etc.
|
|
However, circlogd supports udp/tcp sockets easily, so that should not be an issue.
|
|
|
|
### syslog-ng
|
|
|
|
To integrate into syslog-ng, put this in `/etc/syslog-ng/conf.d/circolog.conf`
|
|
```
|
|
destination d_circolog {
|
|
unix-dgram("/run/circolog-syslog.sock"
|
|
flags(syslog-protocol)
|
|
);
|
|
};
|
|
log { source(s_src); destination(d_circolog); };
|
|
```
|
|
and run `circologd -syslogd-socket /run/circolog-syslog.sock -query-socket /run/circolog-query.sock`
|
|
|
|
|
|
## Client
|
|
|
|
`circolog` has its own client: `circolog-tail`. It is intended to resemble `tail -f` for the most basic
|
|
options; however, it will include filtering options that are common when you want to read logs, because that's
|
|
very easy when you have structured logs available.
|
|
|
|
However, one design point of circolog is to be usable without having a specific client: so the logs are
|
|
offered on both HTTP and websocket. This means that you can use `curl` if you want:
|
|
|
|
curl --unix-socket /run/circolog-query.sock localhost/
|
|
|
|
will give you everything that circologd has in memory.
|
|
|
|
If you want to "follow" (as in `tail -f`) you need to use the websocket interface. However, I don't know of
|
|
any websocket client supporting UNIX domain socket, so you have two options:
|
|
|
|
1. Use `circolog-tail`
|
|
2. Use `circologd` with `-query-addr 127.0.0.1:9080`, add some iptables rule to prevent non-root to access that
|
|
port, and run `ws ws://localhost:9080/ws`. You'll get all the "backlog", and will follow new log messages.
|
|
|
|
### HTTP URLs and parameters
|
|
|
|
When using HTTP, logs are served on `/`. Valid parameters are:
|
|
|
|
* `l`. This is the amount of lines to send. This is essentially the same as the `-n` parameter on tail
|
|
Using `l=-1` (the default) means "give me every log message that you have
|
|
* `fmt`. This selects the output format. When `fmt=json` is used, each message is returned as JSON structured
|
|
data. The format of those JSON messages is still unstable. `fmt=syslog`, the default, outputs messages using "syslog style" (RFC XXXXXX)
|
|
|
|
To use websocket, request path `/ws`. The same parameters of `/` are recognized.
|
|
|
|
## Control daemon
|
|
|
|
Circologd can be controlled, on some aspects, at run-time. It has 2 mechanisms for that: the easiest, and more
|
|
limited, is sending a signal with kill; the second, and more powerful, is a control socket, where you can give
|
|
commands to it. This control socket is just HTTP, so again `curl` is your friend. In the future a
|
|
`circolog-ctl` client will be developed.
|
|
|
|
### Pause
|
|
|
|
When circologd is paused, every new message it receives is immediately discarded. No exception. The backlog
|
|
is, however, preserved. This means that you can trigger the event that you want to investigate, pause
|
|
circolog, then analyze the logs.
|
|
Pausing might be the easiest way to make circologd only run "when needed".
|
|
|
|
When circologd resumes, no previous message is lost.
|
|
|
|
To pause/unpause:
|
|
* `circologctl pause`
|
|
* `pkill -USR1 circologd`
|
|
* `POST /pause/toggle` to your circologd control socket
|
|
|
|
### Clear
|
|
|
|
When you clear the circologd's buffer, it will discard every message it has, but will keep collecting new
|
|
messages.
|
|
|
|
You can do that with `POST /logs/clear`
|
|
|
|
### Filter
|
|
|
|
circologd can drop irrelevant messages using filters. A filter is a sql-like expression (for the exact syntax
|
|
you can see [the doc for the underlying library](https://github.com/araddon/qlbridge/blob/master/FilterQL.md),
|
|
qlbridge), but just imitating sql where clauses can be enough!
|
|
|
|
`circologctl filter message NOT LIKE '%usb%'` will discard everything related to usb.
|
|
|
|
The filter will be applied to incoming messages, so messages mentioning usb will not be saved in memory at all.
|
|
|
|
You can put zero or one filters at a time. That is, you can not stack more filters... but FilterQL syntax
|
|
supports `AND` operators, so this is not an issue.
|
|
|
|
To remove filtering (thus accepting every message) run `circologctl filter`
|
|
|
|
NOTE: `circolog-tail` supports filters with exactly the same syntax, but they are two different kinds of
|
|
filtering: one is server-side, the other is client-side. When you filter server-side with `circologctl
|
|
filter`, circologd will refuse messages not matching the filter. If you only filter with `circolog-tail`, the
|
|
message you are filtering out will still consume space in memory (and will be available to other clients).
|
|
|
|
Filtering brings big dependencies, which will add some 5-6 megabytes to circolog binaries. If you want to
|
|
avoid it, install with `go install -tags nofilter git.lattuga.net/boyska/circolog/...` and your binaries will
|
|
be a bit smaller.
|