Merge branch 'master' of https://git.lattuga.net/boyska/Feedati

this is a big change: OpenRC introduced to tt-rss container.
However, this also means that environment variables set in
docker-compose.yml are not passed to apache.

.gitignore

@@ -0,0 +1,2 @@
+.doit.db
+__pycache__/

INSTALL.Debian.md

@@ -0,0 +1,16 @@
+Feedati has been tested under debian stretch only. It will receive some test under debian buster, too. Don't
+expect much more than this.
+
+If using stretch, add stretch-backports to your repositories
+
+    echo deb http://ftp.debian.org/debian stretch-backports main >> /etc/apt/sources.list.d/backports.list
+
+add docker repositories following
+https://docs.docker.com/install/linux/docker-ce/debian/#set-up-the-repository
+
+    apt-get update
+    apt-get install docker.io golang-go python3-pip python3-doit
+    go get github.com/betalo-sweden/await
+    pip3 install docker-compose
+
+ready!

Makefile

@@ -1,48 +0,0 @@
-update:
-	git submodule update
-
-fix-permissions:
-	chmod -R 777 tt-rss/feed-icons/ tt-rss/cache/ tt-rss/lock/
-
-build: fix-permissions
-	docker-compose build
-	docker-compose up -d
-	$(eval DB_ID := $(shell docker ps -qf name=feedati_db))
-	@echo "[build] DB_ID = $(DB_ID)"
-	await -t 5m postgres://ttrss:password-dev@localhost:5432/ttrss -- \
-		docker cp ./docker/ttrss.sql $$(docker ps -qf name=feedati_db):/tmp/ttrss.sql
-	docker exec -t $$(docker ps -qf name=feedati_db) su -c "psql -d ttrss < /tmp/ttrss.sql" postgres
-	docker exec -t $$(docker ps -qf name=feedati_db) rm -f /tmp/ttrss.sql
-	docker-compose stop
-
-rebuild-tt-rss:
-	docker-compose build tt-rss
-
-rebuild-rss-bridge:
-	docker-compose build rss-bridge
-
-rebuild: rebuild-tt-rss rebuild-rss-bridge
-
-run: fix-permissions
-	docker-compose up
-
-clean:
-	docker container rm $$(docker container ls -a|grep feedati|awk '{print $$1}')
-
-clean-all: clean
-	docker rmi $$(docker images|grep feedati|awk '{print $3}') || true
-	docker volume rm feedati_postgres_data || true
-
-update-tt-rss:
-	cd tt-rss \
-	&& git reset \
-	&& git pull origin master \
-	&& git push lattuga master
-
-update-rss-bridge:
-	cd rss-bridge \
-	&& git reset \
-	&& git pull origin master \
-	&& git push lattuga master
-
-update-remotes: update-rss-bridge update-tt-rss

README.md

@@ -7,25 +7,12 @@ La nuova frontiera dello smanettamento con gli RSS
 
 Serve installare le seguenti cosucce:
 
- - GNU make
  - docker
  - docker-compose
+ - `go get 'github.com/betalo-sweden/await'` (serve Go >= 1.8)
+ - [doit](http://pydoit.org/)  (`pip3 install doit` or `apt-get install python3-doit`)
 
-## Configurami
+## Lanciare
 
-Ci pensa il Makefile:
+Running `doit` is enough.
-
+It will build images, configure them, then run everything.
-```
-make build
-```
-
-## Giocare
-
-Ogni volta che si vuole giocare:
-
-```
-make run
-```
-
-Per fermare il giochino dai un bel ctrl+c. Puoi riprendere
-in qualsiasi momento con `make run`.

doc/GIT.md

@@ -0,0 +1,21 @@
+# Using git
+
+## Howto
+
+## Guidelines
+
+### Commit message
+
+You can drink&code, but you can't drink&git!
+
+    git config --local commit.template doc/gitmessage
+
+As the message explains, please add a prefix to your commit showing the "subject" of your commit.
+If your commit covers too many topic, it might be that it is just wrong.
+However, sometimes it is fine: for example, if you write some PHP code on tt-rss which adds a new dependency
+on some PHP extension, you will need to add that dependency into Dockerfile-tt-rss. So, both [D] and [TT]
+could apply.
+When multiple subjects apply, choose which is more important to you. 2 prefixes are allowed, if you really
+think so.
+Remember: if we wanted to know what changed in your commit, we would use git log --stat. The commit prefix
+should catch the "spirit" of the commit.

doc/HACKING.md

@@ -0,0 +1,33 @@
+The instructions in README.md are fine when you only want to run. However, when developing, you might need to
+know some more tricks.
+
+The goal of our build manager (`dodo.py`) is to make a single call to `doit`, without arguments, always a sane
+and correct choice. However that could not always be the case.
+
+Let's organize our explanation into "iteration loops": you are typically developing on some aspect of it, and
+not on the entire setup. Based on this, you'll need to do different things.
+
+## Loops
+
+### Hacking code on tt-rss or rss-bridge
+
+If you are changing PHP code, well, this is the easiest part.
+Run `doit`, and you should have everything running, and logs showing up.
+Then, with your favourite editor, edit the PHP files you prefer. You can then refresh what you want in the
+browser, and everything should be automatically changing. No other action required!
+
+### Changing nginx configuration
+
+Currently, there is no way to reload nginx, even though it should not be impossible, theoretically, to do it.
+So edit `docker/nginx.conf` with your favourite editor then run `doit`. On the next change, give `ctrl-c` to
+`doit` and then again run `doit`.
+
+This workflow could be improved, being currently a bit too slow for such an easy thing.
+
+### Change some docker file
+
+When you change the dockerfile you need to build everything again. Doit should notice this, and just running
+`doit` should build everything again. If for some reason this is not the case, `doit clean build` will clean
+all of our artifacts; so that running `doit` will surely rebuild everything.
+
+Notice that building is a bit slow. There is no way to improve this, so just keep it in mind!

doc/gitmessage

@@ -0,0 +1,8 @@
+
+
+# [D] = differences in docker{,-compose}; [TT] = tt-rss; [BR] = rss-bridge
+# First line is WHY (50chars), should complete the sentence
+#    If applied, this commit will...
+# Body is WHAT
+# Never explain how: the diff itself will
+# Best practice: http://chris.beams.io/posts/git-commit/

docker-compose.yml

@@ -20,11 +20,17 @@ services:
             dockerfile: ./docker/Dockerfile-tt-rss
         container_name: feedati_tt_rss
         volumes:
-            - ./tt-rss:/var/www/html
+            - ./tt-rss:/app/public/tt-rss/
-            - ./docker/ttrss-config.php:/var/www/html/config.php:ro
+            - ./docker/ttrss-config.php:/app/public/tt-rss/config.php:ro
         ports:
             - 8000:80
             - 9312:9312
+        environment:
+            - APACHE_SERVER_NAME=tt-rss
+            - PHP_DISPLAY_ERRORS=stdout
+            - PHP_DISPLAY_STARTUP_ERROR=1
+        depends_on:
+            - db
 
     rss-bridge:
         image: feedati/rss-bridge:latest
@@ -43,6 +49,9 @@ services:
             - ./docker/nginx.conf:/etc/nginx/conf.d/default.conf:ro
         ports:
             - 80:80
+        depends_on:
+            - tt-rss
+            - rss-bridge
 
 volumes:
     postgres_data:

docker/Dockerfile-tt-rss

@@ -1,10 +1,30 @@
-FROM php:7.2-apache
+FROM ulsmith/alpine-apache-php7
+RUN apk add --update php7-pgsql php7-fileinfo openrc postgresql-client && apk del php7-pdo_mysql php7-pdo_odbc php7-pdo_sqlite db php7-redis php7-ftp && rm -rf /var/cache/apk/
+RUN mkdir -p /app/public/tt-rss/ /var/cache/tt-rss/images \
+ /var/cache/tt-rss/upload /var/cache/tt-rss/export/ \
+ /var/cache/tt-rss/js /var/lock/tt-rss/ && \
+ chown -R apache:root /app/public /var/cache/tt-rss/ /var/lock/tt-rss/ && \
+# thanks https://github.com/neeravkumar/dockerfiles/blob/master/alpine-openrc/Dockerfile
+# Tell openrc its running inside a container, till now that has meant LXC
+ sed -i 's/#rc_sys=""/rc_sys="lxc"/g' /etc/rc.conf &&\
+# Tell openrc loopback and net are already there, since docker handles the networking
+    echo 'rc_provide="loopback net"' >> /etc/rc.conf &&\
+# can't get ttys unless you run the container in privileged mode
+    sed -i '/tty/d' /etc/inittab &&\
+# can't set hostname since docker sets it
+    sed -i 's/hostname $opts/# hostname $opts/g' /etc/init.d/hostname &&\
+# can't mount tmpfs since not privileged
+    sed -i 's/mount -t tmpfs/# mount -t tmpfs/g' /lib/rc/sh/init.sh &&\
+# can't do cgroups
+sed -i 's/cgroup_add_service$/# cgroup_add_service /g' /lib/rc/sh/openrc-run.sh
 
-ENV DEBIAN_FRONTEND=noninteractive
+COPY tt-rss/* /app/public/tt-rss/
+COPY docker/ttrss-openrc-apache /etc/init.d/apache
+COPY docker/ttrss-openrc-ttrssupdate /etc/init.d/ttrssupdate
+RUN rc-update add apache && rc-update add ttrssupdate
 
-RUN apt-get update && apt-get install --no-install-recommends -y libpq-dev \
+
-    && docker-php-ext-configure pgsql -with-pgsql=/usr/local/pgsql \
+ENTRYPOINT []
-    && docker-php-ext-install pdo pgsql pdo_pgsql \
+CMD ["/sbin/init"]
-    && docker-php-ext-enable pdo pgsql pdo_pgsql \
+
- && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+# vim: set ft=dockerfile:
-COPY tt-rss/* /var/www/html/

docker/nginx.conf

@@ -8,7 +8,7 @@ server {
 
     location /tt-rss {
         resolver              127.0.0.11 ipv6=off;
-        proxy_pass            http://tt-rss/;
+        proxy_pass            http://tt-rss/tt-rss;
         proxy_read_timeout    60s;
         proxy_connect_timeout 4s;
         proxy_set_header      Host $host;

docker/ttrss-config.php

@@ -18,7 +18,7 @@
 	// *** Basic settings (important!) ***
 	// ***********************************
 
-	define('SELF_URL_PATH', 'http://localhost');
+	define('SELF_URL_PATH', 'http://localhost/tt-rss/');
 	// Full URL of your tt-rss installation. This should be set to the
 	// location of tt-rss directory, e.g. http://example.org/tt-rss/
 	// You need to set this option correctly otherwise several features
@@ -61,11 +61,11 @@
 	// then most probably you are using the CGI binary. If you are unsure what to
 	// put in here, ask your hosting provider.
 
-	define('LOCK_DIRECTORY', 'lock');
+	define('LOCK_DIRECTORY', '/var/lock/tt-rss');
 	// Directory for lockfiles, must be writable to the user you run
 	// daemon process or cronjobs under.
 
-	define('CACHE_DIR', 'cache');
+	define('CACHE_DIR', '/var/cache/tt-rss');
 	// Local cache directory for RSS feed content.
 
 	define('ICONS_DIR', "feed-icons");
@@ -196,4 +196,4 @@
 	// Expected config version. Please update this option in config.php
 	// if necessary (after migrating all new options from this file).
 
-	// vim:ft=php
+	// vim:ft=php bkc=yes:

docker/ttrss-openrc-apache

@@ -0,0 +1,18 @@
+#!/sbin/runscript
+
+start() {
+    ebegin "Starting Apache (with wrapper)"
+    start-stop-daemon --background --start --exec /bootstrap/start.sh \
+        --make-pidfile --pidfile /var/run/apache-start.pid
+    eend $?
+}
+
+
+stop() {
+    ebegin "Stopping Apache (with wrapper)"
+    start-stop-daemon --stop --exec /bootstrap/start.sh \
+        --pidfile /var/run/apache-start.pid
+    start-stop-daemon --stop --exec httpd \
+        --pidfile /var/run/apache2/httpd.pid
+    eend $?
+}

docker/ttrss-openrc-ttrssupdate

@@ -0,0 +1,20 @@
+#!/sbin/openrc-run
+
+name="ttrssupdate"
+description="continously update tt-rss feeds"
+command="/usr/bin/php7"
+command_args="/app/public/tt-rss/update.php --daemon"
+command_user="apache"
+pidfile="/run/$name.pid"
+start_stop_daemon_args=""
+
+start_pre() {
+    for _ in $(seq 1 60); do
+        if env PGPASSWORD=password-dev psql -h db -U ttrss -w ttrss; then
+            return 0
+        fi
+        einfo "Waiting... ( $? )"
+        sleep 1
+    done
+    return 1
+}

dodo.py

@@ -0,0 +1,113 @@
+import subprocess
+
+from doit.tools import LongRunning
+
+from dodo_utils import wait_net_service, wait_pgsql_db, \
+        up2date_hasimage, up2date_anyimages, \
+        run_task_func
+
+COMPOSE = 'docker-compose -p feedati'
+DOIT_CONFIG = {'default_tasks': ['up']}
+
+
+def task_build():
+    '''builda il container docker'''
+    return {
+        'uptodate': [up2date_anyimages],
+        'file_dep': ['docker-compose.yml', 'docker/Dockerfile-tt-rss',
+                     'docker/ttrss-openrc-apache',
+                     'docker/ttrss-openrc-ttrssupdate'],
+        'actions': [COMPOSE + ' build'],
+        'clean': [run_task_func(task__build_rm),
+                  run_task_func(task__build_rmi)],
+        'doc': '''
+        This task recreates every docker container. While it is automatically run for most changes in the
+        development environment, please remember that if you want to run it manually to grab changes in the
+        docker hub, you need to run `doit run -a build`.
+        '''
+    }
+
+
+def task__build_rm():
+    '''rimuove container avviati'''
+    return {'actions': [
+        "docker container ls -a --format '{{.ID}}\t{{.Names}}'|"
+        "awk '$2 ~ /^feedati_/ { print  $1 }' | "
+        "xargs -r docker container rm",
+    ]}
+
+
+def task__build_rmi():
+    '''rimuove immagini ottenute con build'''
+    return {
+        'actions': [
+            r"docker images -q 'feedati/*' |"
+            "xargs -r --verbose docker rmi",
+        ]
+    }
+
+
+def task__dbprepare_clean():
+    '''rimuove il dump caricato sul db'''
+    return {
+        'actions': [
+            "docker ps -aqf name=feedati_db|xargs -r docker container rm ",
+
+            "docker volume rm feedati_postgres_data || true",
+        ]
+    }
+
+
+def stop():
+    subprocess.check_call((COMPOSE + ' stop').split())
+    return True
+
+
+def task_dbprepare():
+    '''applica il dump sql al container del db'''
+    return {
+        'setup': ['_dbprepare_clean', 'build'],
+        'file_dep': ['docker/ttrss.sql'],
+        'actions': [
+                    (COMPOSE + ' up -d db').split(),
+                    (wait_net_service, ('localhost', 5432, 300)),
+                    (wait_pgsql_db, ('feedati_db', 'ttrss', 'ttrss')),
+                    'echo LOADING DB',
+                    r'docker exec -i $(docker ps -aqf name=feedati_db) '
+                    'psql -h 127.0.0.1 -f - -d ttrss ttrss < docker/ttrss.sql',
+                    'echo DB RESTORED',
+                    ],
+        'teardown': [(stop, [])],
+        'uptodate': [up2date_hasimage('feedati_postgres_data')()],
+        'clean': [run_task_func(task__dbprepare_clean)]
+    }
+
+
+def task__fix_perms():
+    '''fix permissions for shared www dir'''
+    return {
+        'actions': [
+            'chmod -R 777 tt-rss/feed-icons/ || true'
+        ]
+    }
+
+
+def task_up():
+    '''RUN that stuff!'''
+    return {
+        'task_dep': ['build', 'dbprepare', '_fix_perms'],
+        'teardown': [(stop, [])],
+        'actions': [LongRunning(
+            (COMPOSE + ' up').split(),
+            shell=False)
+        ]
+    }
+
+
+def task__cleanall():
+    '''clean everything there is to clean'''
+    return {
+        'task_dep': ['_build_rm', '_build_rmi', '_dbprepare_clean'],
+        'actions': None
+    }
+

dodo_utils.py

@@ -0,0 +1,99 @@
+import subprocess
+import time
+
+from doit import loader
+
+
+def wait_net_service(server, port, timeout=None):
+    """ Wait for network service to appear 
+        @param timeout: in seconds, if None or 0 wait forever
+        @return: True of False, if timeout is None may return only True or
+                 throw unhandled network exception
+    """
+    import socket
+    import errno
+
+    s = socket.socket()
+    if timeout:
+        from time import time as now
+        # time module is needed to calc timeout shared between two exceptions
+        end = now() + timeout
+
+    while True:
+        try:
+            if timeout:
+                next_timeout = end - now()
+                if next_timeout < 0:
+                    return False
+                else:
+                    s.settimeout(next_timeout)
+            s.connect((server, port))
+        except socket.timeout as err:
+            # this exception occurs only if timeout is set
+            if timeout:
+                return False
+        except socket.error as err:
+            # catch timeout exception from underlying network library
+            # this one is different from socket.timeout
+            if type(err.args) != tuple or err[0] != errno.ETIMEDOUT:
+                raise
+        else:
+            s.close()
+            return True
+
+
+def wait_pgsql_db(container, dbname, user, timeout=60):
+    container_id = subprocess.check_output(
+        'docker ps -qf name=feedati_db'.split()).decode('utf8').strip()
+    print('container db=', container_id)
+    cmd = ['docker', 'exec', container_id,
+           'psql', '-h', 'localhost', '-U', user, '-w', dbname]
+    for i in range(timeout):
+        try:
+            subprocess.check_output(cmd, shell=False)
+        except subprocess.CalledProcessError as exc:
+            if exc.returncode == 2:  # there's some more to wait
+                time.sleep(1)
+                continue
+            raise
+        else:
+            return True
+
+
+def up2date_anyimages():
+    cmd = "docker images -q feedati/*".split()
+    output = subprocess.check_output(cmd).strip()
+    if output:
+        return True
+    return False
+
+
+def up2date_hasimage(imagename):
+    def fun():
+        cmd = ' '.join(['docker', 'volume', 'ls',
+                        '-qf', 'name=%s' % imagename])
+        out = subprocess.check_output(cmd, shell=True).decode('utf8').strip()
+        if out:
+            return True
+        return False
+    return fun
+
+
+def run(cmd, **kwargs):
+    def fun():
+        subprocess.check_call(cmd, **kwargs)
+        return True
+    return fun
+
+
+def run_task_func(taskf):
+    ret = taskf()
+    tasks = loader.generate_tasks(taskf.__name__, ret, taskf.__doc__)
+
+    def fun():
+        for task in tasks:
+            task.execute()
+        return True
+    if tasks:
+        fun.__doc__ = '\n'.join(t.doc for t in tasks)
+    return fun