"http" should be delegated to an external program #20

New issue

Open

opened 2022-05-06 14:10:51 +00:00 by boyska · 1 comment

boyska commented

2022-05-06 14:10:51 +00:00

Owner

Describe what happens

Most HTTP operations are handled in fsutils.py. This is good, but the more feature request we have (#17, #18) the more it is going to become a full-featured HTTP client on its own. It just doesn't scale.

what should happen, instead?

HTTP operations should be handled by proper HTTP clients (ie: wget).

More than this, it should be possible to override this HTTP client. This would allow a sysadmin to add site-specific quirks.

Implementation proposal

program selection

larigira.fsutils.download_http should not call urllib.request.urlretrieve. Rather, it should invoke the first among:

~/.config/larigira/overrides/httpdl
$LARIGIRA_DATA/bin/httpdl

program interface

httpdl will be called with a single argument: the URL to be downloaded.

Any other relevant information will be passed as environment variable (see below for details).

The script must signal its successful completion with a 0 exit code. Any other exit code is considered an error.

The script stdout will be considered to be the filename of the downloaded file.
If the exit code was not 0, stdout is ignored.

Environment

all the environment of larigira will be available to this script
AUDIOSPEC_ID will contain the information for whatever this is part of

Cleanup

Temporary files

The script will be responsible for cleaning up its temporary files, if any.

UnusedCleaner

We shouldn't need to do anything special wrt UnusedCleaner: when the file is added to the playlist, it will also be added to the UnusedCleaner mechanism.

## Describe what happens Most HTTP operations are handled in `fsutils.py`. This is good, but the more feature request we have (#17, #18) the more it is going to become a full-featured HTTP client on its own. It just doesn't scale. ### what should happen, instead? HTTP operations should be handled by proper HTTP clients (ie: wget). More than this, it should be possible to override this HTTP client. This would allow a sysadmin to add site-specific quirks. # Implementation proposal ## program selection `larigira.fsutils.download_http` should not call `urllib.request.urlretrieve`. Rather, it should invoke the first among: 1. `~/.config/larigira/overrides/httpdl` 2. `$LARIGIRA_DATA/bin/httpdl` ## program interface `httpdl` will be called with a single argument: the URL to be downloaded. Any other relevant information will be passed as environment variable (see below for details). The script must signal its successful completion with a `0` exit code. Any other exit code is considered an error. The script stdout will be considered to be the filename of the downloaded file. If the exit code was not `0`, stdout is ignored. ### Environment * all the environment of `larigira` will be available to this script * `AUDIOSPEC_ID` will contain the information for whatever this is part of ### Cleanup #### Temporary files The script will be responsible for cleaning up its temporary files, if any. #### UnusedCleaner We shouldn't need to do anything special wrt UnusedCleaner: when the file is added to the playlist, it will also be added to the UnusedCleaner mechanism.

boyska commented

2025-03-01 01:14:10 +00:00

Author

Owner

Relevant:

 def rewrite(url: str) -> str:
     rewrite_bin = Path(__file__).parent / "lib" / "rewrite-url"
     if rewrite_bin.exists():
         url = check_output([str(rewrite_bin), url], encoding="utf8").split("\n")[0]
     return url

Relevant: https://git.lattuga.net/boyska/larigira-scripts/src/commit/424a8826b339ea7748a13b9300ea157fca6a9659/feed#L528-L532