Compare commits

..

17 commits

Author SHA1 Message Date
4fbc87075d Merge branch 'master' of https://git.lattuga.net/boyska/Feedati 2018-08-10 14:44:37 +02:00
a259eb9635 [d] Fixing startup order for containers 2018-08-10 14:44:28 +02:00
1c78a21eb8 git guidelines 2018-08-10 12:00:20 +02:00
4903435659 docker ttrss: openrc properly kills apache 2018-08-10 10:12:06 +02:00
212791c70a tt-rss updates feeds (introducing OpenRC)
this is a big change: OpenRC introduced to tt-rss container.
However, this also means that environment variables set in
docker-compose.yml are not passed to apache.
2018-08-10 02:51:22 +02:00
efbcbc8204 tt-rss moved to /tt-rss/
also, state dirs are moved to /var/cache/ and /var/lock/, thus avoiding
their unneeded "copy" to host.
ICONS_DIR is still pointing to documentroot, so this is not yet
complete.
2018-08-09 23:51:31 +02:00
ff59b1cf3e FIX editing of ttrss-config.php
vim would break it if you don't set bkc=yes
2018-08-09 17:55:27 +02:00
4279c8f09f ctrl-c while "doit up" works better 2018-08-09 17:55:12 +02:00
56f3127af6 doit: better db handling
wait for db to be really ready;
clearly detect if the db still needs to be prepared
2018-08-09 17:37:25 +02:00
58de37f809 tt-rss running on alpine
same base image as rss-bridge, so that the whole environment is more
coherent
2018-08-09 16:52:03 +02:00
5c531d6d85 doc for devs 2018-08-09 14:01:16 +02:00
07ff5594fe some more install info 2018-08-09 14:01:16 +02:00
1318d88a34 who needs makefiles? 2018-08-09 13:37:09 +02:00
0ff7263d8e doit: reorganization
- default task (up)
 - automatic clean actions: doit clean -a now works!
 - FIX some commands (shell errors in clean actions)
 - private tasks (doit list is not a mess anymore; doit list -p is)
2018-08-09 13:17:39 +02:00
96c225da50 first doit implementation 2018-08-09 10:34:34 +02:00
1a51418002 docker-compose clearer names and parsing 2018-08-09 00:27:33 +02:00
c5a1de63bc await is a dependency 2018-08-09 00:27:33 +02:00
15 changed files with 379 additions and 81 deletions

2
.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
.doit.db
__pycache__/

16
INSTALL.Debian.md Normal file
View file

@ -0,0 +1,16 @@
Feedati has been tested under debian stretch only. It will receive some test under debian buster, too. Don't
expect much more than this.
If using stretch, add stretch-backports to your repositories
echo deb http://ftp.debian.org/debian stretch-backports main >> /etc/apt/sources.list.d/backports.list
add docker repositories following
https://docs.docker.com/install/linux/docker-ce/debian/#set-up-the-repository
apt-get update
apt-get install docker.io golang-go python3-pip python3-doit
go get github.com/betalo-sweden/await
pip3 install docker-compose
ready!

View file

@ -1,48 +0,0 @@
update:
git submodule update
fix-permissions:
chmod -R 777 tt-rss/feed-icons/ tt-rss/cache/ tt-rss/lock/
build: fix-permissions
docker-compose build
docker-compose up -d
$(eval DB_ID := $(shell docker ps -qf name=feedati_db))
@echo "[build] DB_ID = $(DB_ID)"
await -t 5m postgres://ttrss:password-dev@localhost:5432/ttrss -- \
docker cp ./docker/ttrss.sql $$(docker ps -qf name=feedati_db):/tmp/ttrss.sql
docker exec -t $$(docker ps -qf name=feedati_db) su -c "psql -d ttrss < /tmp/ttrss.sql" postgres
docker exec -t $$(docker ps -qf name=feedati_db) rm -f /tmp/ttrss.sql
docker-compose stop
rebuild-tt-rss:
docker-compose build tt-rss
rebuild-rss-bridge:
docker-compose build rss-bridge
rebuild: rebuild-tt-rss rebuild-rss-bridge
run: fix-permissions
docker-compose up
clean:
docker container rm $$(docker container ls -a|grep feedati|awk '{print $$1}')
clean-all: clean
docker rmi $$(docker images|grep feedati|awk '{print $3}') || true
docker volume rm feedati_postgres_data || true
update-tt-rss:
cd tt-rss \
&& git reset \
&& git pull origin master \
&& git push lattuga master
update-rss-bridge:
cd rss-bridge \
&& git reset \
&& git pull origin master \
&& git push lattuga master
update-remotes: update-rss-bridge update-tt-rss

View file

@ -7,25 +7,12 @@ La nuova frontiera dello smanettamento con gli RSS
Serve installare le seguenti cosucce: Serve installare le seguenti cosucce:
- GNU make
- docker - docker
- docker-compose - docker-compose
- `go get 'github.com/betalo-sweden/await'` (serve Go >= 1.8)
- [doit](http://pydoit.org/) (`pip3 install doit` or `apt-get install python3-doit`)
## Configurami ## Lanciare
Ci pensa il Makefile: Running `doit` is enough.
It will build images, configure them, then run everything.
```
make build
```
## Giocare
Ogni volta che si vuole giocare:
```
make run
```
Per fermare il giochino dai un bel ctrl+c. Puoi riprendere
in qualsiasi momento con `make run`.

21
doc/GIT.md Normal file
View file

@ -0,0 +1,21 @@
# Using git
## Howto
## Guidelines
### Commit message
You can drink&code, but you can't drink&git!
git config --local commit.template doc/gitmessage
As the message explains, please add a prefix to your commit showing the "subject" of your commit.
If your commit covers too many topic, it might be that it is just wrong.
However, sometimes it is fine: for example, if you write some PHP code on tt-rss which adds a new dependency
on some PHP extension, you will need to add that dependency into Dockerfile-tt-rss. So, both [D] and [TT]
could apply.
When multiple subjects apply, choose which is more important to you. 2 prefixes are allowed, if you really
think so.
Remember: if we wanted to know what changed in your commit, we would use git log --stat. The commit prefix
should catch the "spirit" of the commit.

33
doc/HACKING.md Normal file
View file

@ -0,0 +1,33 @@
The instructions in README.md are fine when you only want to run. However, when developing, you might need to
know some more tricks.
The goal of our build manager (`dodo.py`) is to make a single call to `doit`, without arguments, always a sane
and correct choice. However that could not always be the case.
Let's organize our explanation into "iteration loops": you are typically developing on some aspect of it, and
not on the entire setup. Based on this, you'll need to do different things.
## Loops
### Hacking code on tt-rss or rss-bridge
If you are changing PHP code, well, this is the easiest part.
Run `doit`, and you should have everything running, and logs showing up.
Then, with your favourite editor, edit the PHP files you prefer. You can then refresh what you want in the
browser, and everything should be automatically changing. No other action required!
### Changing nginx configuration
Currently, there is no way to reload nginx, even though it should not be impossible, theoretically, to do it.
So edit `docker/nginx.conf` with your favourite editor then run `doit`. On the next change, give `ctrl-c` to
`doit` and then again run `doit`.
This workflow could be improved, being currently a bit too slow for such an easy thing.
### Change some docker file
When you change the dockerfile you need to build everything again. Doit should notice this, and just running
`doit` should build everything again. If for some reason this is not the case, `doit clean build` will clean
all of our artifacts; so that running `doit` will surely rebuild everything.
Notice that building is a bit slow. There is no way to improve this, so just keep it in mind!

8
doc/gitmessage Normal file
View file

@ -0,0 +1,8 @@
# [D] = differences in docker{,-compose}; [TT] = tt-rss; [BR] = rss-bridge
# First line is WHY (50chars), should complete the sentence
# If applied, this commit will...
# Body is WHAT
# Never explain how: the diff itself will
# Best practice: http://chris.beams.io/posts/git-commit/

View file

@ -20,11 +20,17 @@ services:
dockerfile: ./docker/Dockerfile-tt-rss dockerfile: ./docker/Dockerfile-tt-rss
container_name: feedati_tt_rss container_name: feedati_tt_rss
volumes: volumes:
- ./tt-rss:/var/www/html - ./tt-rss:/app/public/tt-rss/
- ./docker/ttrss-config.php:/var/www/html/config.php:ro - ./docker/ttrss-config.php:/app/public/tt-rss/config.php:ro
ports: ports:
- 8000:80 - 8000:80
- 9312:9312 - 9312:9312
environment:
- APACHE_SERVER_NAME=tt-rss
- PHP_DISPLAY_ERRORS=stdout
- PHP_DISPLAY_STARTUP_ERROR=1
depends_on:
- db
rss-bridge: rss-bridge:
image: feedati/rss-bridge:latest image: feedati/rss-bridge:latest
@ -43,6 +49,9 @@ services:
- ./docker/nginx.conf:/etc/nginx/conf.d/default.conf:ro - ./docker/nginx.conf:/etc/nginx/conf.d/default.conf:ro
ports: ports:
- 80:80 - 80:80
depends_on:
- tt-rss
- rss-bridge
volumes: volumes:
postgres_data: postgres_data:

View file

@ -1,10 +1,30 @@
FROM php:7.2-apache FROM ulsmith/alpine-apache-php7
RUN apk add --update php7-pgsql php7-fileinfo openrc postgresql-client && apk del php7-pdo_mysql php7-pdo_odbc php7-pdo_sqlite db php7-redis php7-ftp && rm -rf /var/cache/apk/
RUN mkdir -p /app/public/tt-rss/ /var/cache/tt-rss/images \
/var/cache/tt-rss/upload /var/cache/tt-rss/export/ \
/var/cache/tt-rss/js /var/lock/tt-rss/ && \
chown -R apache:root /app/public /var/cache/tt-rss/ /var/lock/tt-rss/ && \
# thanks https://github.com/neeravkumar/dockerfiles/blob/master/alpine-openrc/Dockerfile
# Tell openrc its running inside a container, till now that has meant LXC
sed -i 's/#rc_sys=""/rc_sys="lxc"/g' /etc/rc.conf &&\
# Tell openrc loopback and net are already there, since docker handles the networking
echo 'rc_provide="loopback net"' >> /etc/rc.conf &&\
# can't get ttys unless you run the container in privileged mode
sed -i '/tty/d' /etc/inittab &&\
# can't set hostname since docker sets it
sed -i 's/hostname $opts/# hostname $opts/g' /etc/init.d/hostname &&\
# can't mount tmpfs since not privileged
sed -i 's/mount -t tmpfs/# mount -t tmpfs/g' /lib/rc/sh/init.sh &&\
# can't do cgroups
sed -i 's/cgroup_add_service$/# cgroup_add_service /g' /lib/rc/sh/openrc-run.sh
ENV DEBIAN_FRONTEND=noninteractive COPY tt-rss/* /app/public/tt-rss/
COPY docker/ttrss-openrc-apache /etc/init.d/apache
COPY docker/ttrss-openrc-ttrssupdate /etc/init.d/ttrssupdate
RUN rc-update add apache && rc-update add ttrssupdate
RUN apt-get update && apt-get install --no-install-recommends -y libpq-dev \
&& docker-php-ext-configure pgsql -with-pgsql=/usr/local/pgsql \ ENTRYPOINT []
&& docker-php-ext-install pdo pgsql pdo_pgsql \ CMD ["/sbin/init"]
&& docker-php-ext-enable pdo pgsql pdo_pgsql \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # vim: set ft=dockerfile:
COPY tt-rss/* /var/www/html/

View file

@ -8,7 +8,7 @@ server {
location /tt-rss { location /tt-rss {
resolver 127.0.0.11 ipv6=off; resolver 127.0.0.11 ipv6=off;
proxy_pass http://tt-rss/; proxy_pass http://tt-rss/tt-rss;
proxy_read_timeout 60s; proxy_read_timeout 60s;
proxy_connect_timeout 4s; proxy_connect_timeout 4s;
proxy_set_header Host $host; proxy_set_header Host $host;

View file

@ -18,7 +18,7 @@
// *** Basic settings (important!) *** // *** Basic settings (important!) ***
// *********************************** // ***********************************
define('SELF_URL_PATH', 'http://localhost'); define('SELF_URL_PATH', 'http://localhost/tt-rss/');
// Full URL of your tt-rss installation. This should be set to the // Full URL of your tt-rss installation. This should be set to the
// location of tt-rss directory, e.g. http://example.org/tt-rss/ // location of tt-rss directory, e.g. http://example.org/tt-rss/
// You need to set this option correctly otherwise several features // You need to set this option correctly otherwise several features
@ -61,11 +61,11 @@
// then most probably you are using the CGI binary. If you are unsure what to // then most probably you are using the CGI binary. If you are unsure what to
// put in here, ask your hosting provider. // put in here, ask your hosting provider.
define('LOCK_DIRECTORY', 'lock'); define('LOCK_DIRECTORY', '/var/lock/tt-rss');
// Directory for lockfiles, must be writable to the user you run // Directory for lockfiles, must be writable to the user you run
// daemon process or cronjobs under. // daemon process or cronjobs under.
define('CACHE_DIR', 'cache'); define('CACHE_DIR', '/var/cache/tt-rss');
// Local cache directory for RSS feed content. // Local cache directory for RSS feed content.
define('ICONS_DIR', "feed-icons"); define('ICONS_DIR', "feed-icons");
@ -196,4 +196,4 @@
// Expected config version. Please update this option in config.php // Expected config version. Please update this option in config.php
// if necessary (after migrating all new options from this file). // if necessary (after migrating all new options from this file).
// vim:ft=php // vim:ft=php bkc=yes:

18
docker/ttrss-openrc-apache Executable file
View file

@ -0,0 +1,18 @@
#!/sbin/runscript
start() {
ebegin "Starting Apache (with wrapper)"
start-stop-daemon --background --start --exec /bootstrap/start.sh \
--make-pidfile --pidfile /var/run/apache-start.pid
eend $?
}
stop() {
ebegin "Stopping Apache (with wrapper)"
start-stop-daemon --stop --exec /bootstrap/start.sh \
--pidfile /var/run/apache-start.pid
start-stop-daemon --stop --exec httpd \
--pidfile /var/run/apache2/httpd.pid
eend $?
}

20
docker/ttrss-openrc-ttrssupdate Executable file
View file

@ -0,0 +1,20 @@
#!/sbin/openrc-run
name="ttrssupdate"
description="continously update tt-rss feeds"
command="/usr/bin/php7"
command_args="/app/public/tt-rss/update.php --daemon"
command_user="apache"
pidfile="/run/$name.pid"
start_stop_daemon_args=""
start_pre() {
for _ in $(seq 1 60); do
if env PGPASSWORD=password-dev psql -h db -U ttrss -w ttrss; then
return 0
fi
einfo "Waiting... ( $? )"
sleep 1
done
return 1
}

113
dodo.py Normal file
View file

@ -0,0 +1,113 @@
import subprocess
from doit.tools import LongRunning
from dodo_utils import wait_net_service, wait_pgsql_db, \
up2date_hasimage, up2date_anyimages, \
run_task_func
COMPOSE = 'docker-compose -p feedati'
DOIT_CONFIG = {'default_tasks': ['up']}
def task_build():
'''builda il container docker'''
return {
'uptodate': [up2date_anyimages],
'file_dep': ['docker-compose.yml', 'docker/Dockerfile-tt-rss',
'docker/ttrss-openrc-apache',
'docker/ttrss-openrc-ttrssupdate'],
'actions': [COMPOSE + ' build'],
'clean': [run_task_func(task__build_rm),
run_task_func(task__build_rmi)],
'doc': '''
This task recreates every docker container. While it is automatically run for most changes in the
development environment, please remember that if you want to run it manually to grab changes in the
docker hub, you need to run `doit run -a build`.
'''
}
def task__build_rm():
'''rimuove container avviati'''
return {'actions': [
"docker container ls -a --format '{{.ID}}\t{{.Names}}'|"
"awk '$2 ~ /^feedati_/ { print $1 }' | "
"xargs -r docker container rm",
]}
def task__build_rmi():
'''rimuove immagini ottenute con build'''
return {
'actions': [
r"docker images -q 'feedati/*' |"
"xargs -r --verbose docker rmi",
]
}
def task__dbprepare_clean():
'''rimuove il dump caricato sul db'''
return {
'actions': [
"docker ps -aqf name=feedati_db|xargs -r docker container rm ",
"docker volume rm feedati_postgres_data || true",
]
}
def stop():
subprocess.check_call((COMPOSE + ' stop').split())
return True
def task_dbprepare():
'''applica il dump sql al container del db'''
return {
'setup': ['_dbprepare_clean', 'build'],
'file_dep': ['docker/ttrss.sql'],
'actions': [
(COMPOSE + ' up -d db').split(),
(wait_net_service, ('localhost', 5432, 300)),
(wait_pgsql_db, ('feedati_db', 'ttrss', 'ttrss')),
'echo LOADING DB',
r'docker exec -i $(docker ps -aqf name=feedati_db) '
'psql -h 127.0.0.1 -f - -d ttrss ttrss < docker/ttrss.sql',
'echo DB RESTORED',
],
'teardown': [(stop, [])],
'uptodate': [up2date_hasimage('feedati_postgres_data')()],
'clean': [run_task_func(task__dbprepare_clean)]
}
def task__fix_perms():
'''fix permissions for shared www dir'''
return {
'actions': [
'chmod -R 777 tt-rss/feed-icons/ || true'
]
}
def task_up():
'''RUN that stuff!'''
return {
'task_dep': ['build', 'dbprepare', '_fix_perms'],
'teardown': [(stop, [])],
'actions': [LongRunning(
(COMPOSE + ' up').split(),
shell=False)
]
}
def task__cleanall():
'''clean everything there is to clean'''
return {
'task_dep': ['_build_rm', '_build_rmi', '_dbprepare_clean'],
'actions': None
}

99
dodo_utils.py Normal file
View file

@ -0,0 +1,99 @@
import subprocess
import time
from doit import loader
def wait_net_service(server, port, timeout=None):
""" Wait for network service to appear
@param timeout: in seconds, if None or 0 wait forever
@return: True of False, if timeout is None may return only True or
throw unhandled network exception
"""
import socket
import errno
s = socket.socket()
if timeout:
from time import time as now
# time module is needed to calc timeout shared between two exceptions
end = now() + timeout
while True:
try:
if timeout:
next_timeout = end - now()
if next_timeout < 0:
return False
else:
s.settimeout(next_timeout)
s.connect((server, port))
except socket.timeout as err:
# this exception occurs only if timeout is set
if timeout:
return False
except socket.error as err:
# catch timeout exception from underlying network library
# this one is different from socket.timeout
if type(err.args) != tuple or err[0] != errno.ETIMEDOUT:
raise
else:
s.close()
return True
def wait_pgsql_db(container, dbname, user, timeout=60):
container_id = subprocess.check_output(
'docker ps -qf name=feedati_db'.split()).decode('utf8').strip()
print('container db=', container_id)
cmd = ['docker', 'exec', container_id,
'psql', '-h', 'localhost', '-U', user, '-w', dbname]
for i in range(timeout):
try:
subprocess.check_output(cmd, shell=False)
except subprocess.CalledProcessError as exc:
if exc.returncode == 2: # there's some more to wait
time.sleep(1)
continue
raise
else:
return True
def up2date_anyimages():
cmd = "docker images -q feedati/*".split()
output = subprocess.check_output(cmd).strip()
if output:
return True
return False
def up2date_hasimage(imagename):
def fun():
cmd = ' '.join(['docker', 'volume', 'ls',
'-qf', 'name=%s' % imagename])
out = subprocess.check_output(cmd, shell=True).decode('utf8').strip()
if out:
return True
return False
return fun
def run(cmd, **kwargs):
def fun():
subprocess.check_call(cmd, **kwargs)
return True
return fun
def run_task_func(taskf):
ret = taskf()
tasks = loader.generate_tasks(taskf.__name__, ret, taskf.__doc__)
def fun():
for task in tasks:
task.execute()
return True
if tasks:
fun.__doc__ = '\n'.join(t.doc for t in tasks)
return fun