README.md 5.59 KB
Newer Older
1
# Log MatomNom (for Matomo 4.x)
Michał Woźniak's avatar
Michał Woźniak committed
2

Michał Woźniak's avatar
Michał Woźniak committed
3
4
5
Matomo's [log analytics](https://matomo.org/log-analytics/), but automatically watching specified directories for logs to ingest. Any detected logfiles are ingested and moved out of the way or deleted.

The script should be useful in situations where webserver logfiles are to be ingested into Matomo upon log rotation, without the need to set up elaborate systems of cronjobs.
Michał Woźniak's avatar
Michał Woźniak committed
6

7
## Requirements
Michał Woźniak's avatar
Michał Woźniak committed
8

9
The `matomnom.py` script requires Matomo's `import_logs.py` (branch `4-x.dev`) log analytics script to be available for import. Since that script only runs on Python 3.x, so does this. Obviously requirements of the `import_logs.py` script need to be satisfied, plus `inotify_simple` and `signal` modules need to be available.
Michał Woźniak's avatar
Michał Woźniak committed
10

11
## Operation
Michał Woźniak's avatar
Michał Woźniak committed
12

13
The script sets [inotify](https://en.wikipedia.org/wiki/Inotify) watches on the listed directories.
Michał Woźniak's avatar
Michał Woźniak committed
14

15
When files matching the `--logfiles-glob` pattern are detected, the script waits `--ingestion-grace-period` seconds after all activity stops and starts ingesting the batch of detected files one by one. Ingested files are either renamed (using `--prefix-ingested` and `--suffix-ingested`) or deleted (`--delete-ingested`).
Michał Woźniak's avatar
Michał Woźniak committed
16

17
If an unrecoverable error occurs during ingestion of a file, the file is either renamed (using `--prefix-failed` and `--suffix-failed`) or deleted (`--delete-failed`) — unless `--exit-on-error` is used, in which case the script immediately exits with an error message.
Michał Woźniak's avatar
Michał Woźniak committed
18

19
20
21
22
23
24
After all files in the batch are processed (either ingested or failed), Matomo's report processing is automagically triggered, unless `--no-auto-archive` is used.

While ingestion is in progress new files are *not* being added to the batch. Once processing of a batch ends, if there are any inotify events since processing started, all files matching the configured glob are added to a new batch, which is then processed. If there are no inotify events since processing of the batch started, script waits for new events.

## Usage

25
Run `./matomnom.py --help` to get help. All `import_logs.py` options are supported, plus these additional ones:
26

Michał Woźniak's avatar
Michał Woźniak committed
27
 - `--logfiles-glob` (default: `"*.log"`)  
28
29
30
31
   Only files matching this shell glob expression will be ingested. It's 
   important to make sure that the glob does not match ingested files after 
   prefix and suffix is applied! See `--prefix-ingested` and `--suffix-ingested`.
        
Michał Woźniak's avatar
Michał Woźniak committed
32
 - `--ingestion-grace-period` (default: `5`)  
33
34
35
36
   Delay (in seconds; fractions are supported) between noticing a logfile to be processed and starting ingesting it.
   This is part of the built-in heuristic for determining that a file is not being modified 
   or moved anymore and can be safely ingested.
        
Michał Woźniak's avatar
Michał Woźniak committed
37
 - `--delete-ingested` (default: False)  
38
39
   Delete successfully ingested logfiles.
        
Michał Woźniak's avatar
Michał Woźniak committed
40
 - `--prefix-ingested` (default: `"ingested/"`)  
41
42
43
44
45
46
47
   Rename ingested logfiles using this prefix; prefix can indicate directories (in 
   which case it should contain '/'), and is then relative to the directory a given 
   logfile was originally in: when watching several directories, a prefix of 
   'ingested/' will place ingested files in './ingested/' subdirectories of 
   respective watched directories. Directories will be created if needed. This option 
   is ignored if `--delete-ingested` is used.
        
Michał Woźniak's avatar
Michał Woźniak committed
48
 - `--suffix-ingested` (default: `".ingested"`)  
49
50
51
   Rename ingested logfiles using this suffix; it cannot contain any '/' characters.
   This option is ignored if `--delete-ingested` is used.
        
Michał Woźniak's avatar
Michał Woźniak committed
52
 - `--exit-on-error` (default: False)  
53
54
   Exit when ingestion errors are encountered.
        
Michał Woźniak's avatar
Michał Woźniak committed
55
 - `--delete-failed` (default: False)  
56
57
58
   Delete logfiles which failed to be ingested.
        
        
Michał Woźniak's avatar
Michał Woźniak committed
59
 - `--prefix-failed` (default: `"failed/"`)  
60
61
62
63
64
65
66
67
68
69
   Rename logfiles that failed to be ingested using this prefix; prefix can 
   have directories (in which case it should contain '/'), and is then relative 
   to the directory a given logfile was originally in: when watching several 
   directories, a prefix of 'failed/' will place such files in './failed/' 
   subdirectories of respective watched directories. Directories will be created 
   if needed. This prefix will also be used for files containing information 
   on what error was encountered and at which line.
   
   This option is ignored if `--delete-failed` is used.
        
Michał Woźniak's avatar
Michał Woźniak committed
70
 - `--suffix-failed` (default: `".failed"`)  
71
72
73
   Rename logfiles that failed to be ingested using this suffix; it cannot 
   contain any '/' characters. This option is ignored if `--delete-failed` is used.
        
Michał Woźniak's avatar
Michał Woźniak committed
74
 - `--no-auto-archive` (default: True)  
75
76
77
78
79
   Do not automatically run auto-archiving of Matomo reports. By default 
   auto-archiving is triggered after a batch of logfiles is ingested

## Docker usage

80
Run the image with log directories you want to watch volume-mounted. Specify the options and directories to watch directly as the command (`matomnom.py` is the entrypoint script, and default command is `--help`).
81
82
83
84

### Example docker-compose service

```yaml
85
86
    matomnom:
        build: https://git.rys.io/libre/log-matomnom.git#matomo-4.x
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
        volumes:
            - "/var/log/old/nginx/:/logs/nginx/"
            - "/var/log/old/apache/:/logs/apache/"
        command:
            - --prefix-ingested
            - ingested/
            - --prefix-failed
            - failed/
            - --enable-http-errors
            - --enable-http-redirects
            - --enable-static
            - --enable-bots
            - --logfiles-glob
            - '*.gz'
            - --url
            - https://matomo.example.org
            - --token-auth
            - <matomo_token>
            - /logs/nginx/
            - /logs/apache/
```