Most HTTP operations are handled in fsutils.py. This is good, but the more feature request we have (#17, #18) the more it is going to become a full-featured HTTP client on its own. It just doesn't scale.
what should happen, instead?
HTTP operations should be handled by proper HTTP clients (ie: wget).
More than this, it should be possible to override this HTTP client. This would allow a sysadmin to add site-specific quirks.
Implementation proposal
program selection
larigira.fsutils.download_http should not call urllib.request.urlretrieve. Rather, it should invoke the first among:
~/.config/larigira/overrides/httpdl
$LARIGIRA_DATA/bin/httpdl
program interface
httpdl will be called with a single argument: the URL to be downloaded.
Any other relevant information will be passed as environment variable (see below for details).
The script must signal its successful completion with a 0 exit code. Any other exit code is considered an error.
The script stdout will be considered to be the filename of the downloaded file.
If the exit code was not 0, stdout is ignored.
Environment
all the environment of larigira will be available to this script
AUDIOSPEC_ID will contain the information for whatever this is part of
Cleanup
Temporary files
The script will be responsible for cleaning up its temporary files, if any.
UnusedCleaner
We shouldn't need to do anything special wrt UnusedCleaner: when the file is added to the playlist, it will also be added to the UnusedCleaner mechanism.
## Describe what happens
Most HTTP operations are handled in `fsutils.py`. This is good, but the more feature request we have (#17, #18) the more it is going to become a full-featured HTTP client on its own. It just doesn't scale.
### what should happen, instead?
HTTP operations should be handled by proper HTTP clients (ie: wget).
More than this, it should be possible to override this HTTP client. This would allow a sysadmin to add site-specific quirks.
# Implementation proposal
## program selection
`larigira.fsutils.download_http` should not call `urllib.request.urlretrieve`. Rather, it should invoke the first among:
1. `~/.config/larigira/overrides/httpdl`
2. `$LARIGIRA_DATA/bin/httpdl`
## program interface
`httpdl` will be called with a single argument: the URL to be downloaded.
Any other relevant information will be passed as environment variable (see below for details).
The script must signal its successful completion with a `0` exit code. Any other exit code is considered an error.
The script stdout will be considered to be the filename of the downloaded file.
If the exit code was not `0`, stdout is ignored.
### Environment
* all the environment of `larigira` will be available to this script
* `AUDIOSPEC_ID` will contain the information for whatever this is part of
### Cleanup
#### Temporary files
The script will be responsible for cleaning up its temporary files, if any.
#### UnusedCleaner
We shouldn't need to do anything special wrt UnusedCleaner: when the file is added to the playlist, it will also be added to the UnusedCleaner mechanism.
Describe what happens
Most HTTP operations are handled in
fsutils.py. This is good, but the more feature request we have (#17, #18) the more it is going to become a full-featured HTTP client on its own. It just doesn't scale.what should happen, instead?
HTTP operations should be handled by proper HTTP clients (ie: wget).
More than this, it should be possible to override this HTTP client. This would allow a sysadmin to add site-specific quirks.
Implementation proposal
program selection
larigira.fsutils.download_httpshould not callurllib.request.urlretrieve. Rather, it should invoke the first among:~/.config/larigira/overrides/httpdl$LARIGIRA_DATA/bin/httpdlprogram interface
httpdlwill be called with a single argument: the URL to be downloaded.Any other relevant information will be passed as environment variable (see below for details).
The script must signal its successful completion with a
0exit code. Any other exit code is considered an error.The script stdout will be considered to be the filename of the downloaded file. If the exit code was not
0, stdout is ignored.Environment
larigirawill be available to this scriptAUDIOSPEC_IDwill contain the information for whatever this is part ofCleanup
Temporary files
The script will be responsible for cleaning up its temporary files, if any.
UnusedCleaner
We shouldn't need to do anything special wrt UnusedCleaner: when the file is added to the playlist, it will also be added to the UnusedCleaner mechanism.