Blallo 0dc1188701 Ignore build/ | 5 years ago | |
---|---|---|
cmd | 5 years ago | |
data | 5 years ago | |
docs | 5 years ago | |
filtering | 5 years ago | |
formatter | 5 years ago | |
vendor | 5 years ago | |
.dockerignore | 5 years ago | |
.drone.yml | 6 years ago | |
.gitignore | 5 years ago | |
.gitmodules | 5 years ago | |
Dockerfile | 5 years ago | |
README.md | 5 years ago | |
hub.go | 5 years ago | |
hub_test.go | 5 years ago | |
release.sh | 5 years ago |
A syslog daemon implementing circular buffer, in-memory storage.
This is useful when you want to keep some (heavily detailed) log available, but you don't want to log too many things to disk. Remember: logging is useful, but can be dangerous to your users' privacy!
On your "main" syslog, forward (part of the) messages to this one!
In these examples I'll refer to the usage of UNIX sockets. They are more secure than TCP/UDP sockets because they have file permissions, they can be "masked" using mount namespaces, etc. However, circlogd supports udp/tcp sockets easily, so that should not be an issue.
To integrate into syslog-ng, put this in /etc/syslog-ng/conf.d/circolog.conf
destination d_circolog {
unix-dgram("/run/circolog-syslog.sock"
flags(syslog-protocol)
);
};
log { source(s_src); destination(d_circolog); };
and run circologd -syslogd-socket /run/circolog-syslog.sock -query-socket /run/circolog-query.sock
circolog
has its own client: circolog-tail
. It is intended to resemble tail -f
for the most basic
options; however, it will include filtering options that are common when you want to read logs, because that's
very easy when you have structured logs available.
However, one design point of circolog is to be usable without having a specific client: so the logs are
offered on both HTTP and websocket. This means that you can use curl
if you want:
curl --unix-socket /run/circolog-query.sock localhost/
will give you everything that circologd has in memory.
If you want to "follow" (as in tail -f
) you need to use the websocket interface. However, I don't know of
any websocket client supporting UNIX domain socket, so you have two options:
circolog-tail
circologd
with -query-addr 127.0.0.1:9080
, add some iptables rule to prevent non-root to access that
port, and run ws ws://localhost:9080/ws
. You'll get all the "backlog", and will follow new log messages.When using HTTP, logs are served on /
. Valid parameters are:
l
. This is the amount of lines to send. This is essentially the same as the -n
parameter on tail
Using l=-1
(the default) means "give me every log message that you havefmt
. This selects the output format. When fmt=json
is used, each message is returned as JSON structured
data. The format of those JSON messages is still unstable. fmt=syslog
, the default, outputs messages using "syslog style" (RFC XXXXXX)To use websocket, request path /ws
. The same parameters of /
are recognized.
Circologd can be controlled, on some aspects, at run-time. It has 2 mechanisms for that: the easiest, and more
limited, is sending a signal with kill; the second, and more powerful, is a control socket, where you can give
commands to it. This control socket is just HTTP, so again curl
is your friend. In the future a
circolog-ctl
client will be developed.
When circologd is paused, every new message it receives is immediately discarded. No exception. The backlog is, however, preserved. This means that you can trigger the event that you want to investigate, pause circolog, then analyze the logs. Pausing might be the easiest way to make circologd only run "when needed".
When circologd resumes, no previous message is lost.
To pause/unpause:
circologctl pause
pkill -USR1 circologd
POST /pause/toggle
to your circologd control socketWhen you clear the circologd's buffer, it will discard every message it has, but will keep collecting new messages.
You can do that with POST /logs/clear
circologd can drop irrelevant messages using filters. A filter is a sql-like expression (for the exact syntax you can see the doc for the underlying library, qlbridge), but just imitating sql where clauses can be enough!
circologctl filter message NOT LIKE '%usb%'
will discard everything related to usb.
The filter will be applied to incoming messages, so messages mentioning usb will not be saved in memory at all.
You can put zero or one filters at a time. That is, you can not stack more filters... but FilterQL syntax
supports AND
operators, so this is not an issue.
To remove filtering (thus accepting every message) run circologctl filter
NOTE: circolog-tail
supports filters with exactly the same syntax, but they are two different kinds of
filtering: one is server-side, the other is client-side. When you filter server-side with circologctl
filter
, circologd will refuse messages not matching the filter. If you only filter with circolog-tail
, the
message you are filtering out will still consume space in memory (and will be available to other clients).
Filtering brings big dependencies, which will add some 5-6 megabytes to circolog binaries. If you want to
avoid it, install with go install -tags nofilter git.lattuga.net/boyska/circolog/...
and your binaries will
be a bit smaller.
To tinker with circolog, there is also a Dockerfile
. Simply:
$ docker build -t circolog .
$ docker run -t circolog