+ watchTower
Python 55.2% - JavaScript 18.7% - Java 10.7% - HTML 6.4% - CSS 5.8% - Dockerfile 2.5% - Shell 0.6%
Last update : 01/07/2021 - 08:15
A simple yet powerful watchtower service.
Heartbeat monitoring + event aggregator.
Hearbeat definitions
- The
server is in charge of monitoring services
- When a
service is considered dead, the server then alerts its owner
- A
service is a pulsing agent
- A
pulse is a sign of aliveness
- When a
service pulse the server, it tells him when he will pulse again in the worst case
- If the
service has not pulsed again in time, the server considers the service dead and alerts its owner
- When a
service is no longer required, it must inform the server to avoid a false dead alerts
- An
alert can be a mail, a pushover message, etc.
Pings can also be configured. In that case, the server regularily pings the service via http or ping with or without proxies.
Event definitions
- The
server is in charge or recieving events
- When a
service sends an event, the server stores it
- On demand,
events can be retrieved for consultation
- An event is:
service + message + level + date of registration
Implementation
- HTTP client / server architecture
- All queries protected by password set in HTTP headers under "password"
- Heartbeat protocol:
- Pulse:
POST /{"service":"SERVICE_NAME","alertType":"ALERT_TYPE","alertTarget":"TARGET_NAME","nextIn":EXPECTED_NEXT_HEARTBEAT_IN_SECS}
- Cancel:
DELETE /{"service":"SERVICE_NAME"}
- List:
GET /
- Event protocol:
- Add:
POST /add-event{"service":"SERVICE_NAME", "level":"level", "message":"the message"}
- List:
GET /list-events- optional params:
service (is), from (above), to (below), level (above), message (contains)
Server
./server- python3
- Dependencies:
pip install -r requirements.txt
- Config:
config/config.json: main config (cp config/sample.json config/config.json)config/server.crt, config/server.key: SSL certificate, used only if SSL activated
- Datas:
data
- Deploy:
- Optionnaly define pyenv-virtualenv:
pyenv virtualenv -f 3.6.8 watchTower
- Install dependencies:
pip install --upgrade pip ; pip install -r requirements.txt
- Setup config
- Generate ssl certificates (optional)
- Hook in with upstart (optional,
./server/config/sample.upstart.conf, http://upstart.ubuntu.com/getting-started.html)
- Run:
python server.py
- Docker:
./docker- Config: place server config files in
./docker/config
- Interactive:
cd docker && tools/dockerBuild && tools/dockerRun
- Detached:
cd docker && tools/dockerBuild && tools/dockerRunDetached
Server GUI
https://hostname.com:443/gui
Java Client
./clientJava- Java 7+
- No dependencies
- Run test:
Python Client
./clientPython- python3
- Dependencies:
requests (pip install requests)
- Run test:
python test.py
Javascript client
./clientJavascriptclient.js is a node module- Dependencies:
packages.json -> devDependencies (npm install)
- Run test:
node test.js
- Run test in browser:
browserify test.js -o bundle.js and open test.html in browser
Certificates
- Self signed:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
- Letsencrypt: TODO
TODO
- Put data writing in a queue
- Put alert sending in a queue
- (Way) better events persister
server is in charge of monitoring servicesservice is considered dead, the server then alerts its ownerservice is a pulsing agentpulse is a sign of alivenessservice pulse the server, it tells him when he will pulse again in the worst caseservice has not pulsed again in time, the server considers the service dead and alerts its ownerservice is no longer required, it must inform the server to avoid a false dead alertsalert can be a mail, a pushover message, etc.Pings can also be configured. In that case, the server regularily pings the service via http or ping with or without proxies.- The
serveris in charge or recievingevents - When a
servicesends anevent, the server stores it - On demand,
eventscan be retrieved for consultation - An event is:
service+message+level+date of registration
Implementation
- HTTP client / server architecture
- All queries protected by password set in HTTP headers under "password"
- Heartbeat protocol:
- Pulse:
POST /{"service":"SERVICE_NAME","alertType":"ALERT_TYPE","alertTarget":"TARGET_NAME","nextIn":EXPECTED_NEXT_HEARTBEAT_IN_SECS}
- Cancel:
DELETE /{"service":"SERVICE_NAME"}
- List:
GET /
- Event protocol:
- Add:
POST /add-event{"service":"SERVICE_NAME", "level":"level", "message":"the message"}
- List:
GET /list-events- optional params:
service (is), from (above), to (below), level (above), message (contains)
Server
./server- python3
- Dependencies:
pip install -r requirements.txt
- Config:
config/config.json: main config (cp config/sample.json config/config.json)config/server.crt, config/server.key: SSL certificate, used only if SSL activated
- Datas:
data
- Deploy:
- Optionnaly define pyenv-virtualenv:
pyenv virtualenv -f 3.6.8 watchTower
- Install dependencies:
pip install --upgrade pip ; pip install -r requirements.txt
- Setup config
- Generate ssl certificates (optional)
- Hook in with upstart (optional,
./server/config/sample.upstart.conf, http://upstart.ubuntu.com/getting-started.html)
- Run:
python server.py
- Docker:
./docker- Config: place server config files in
./docker/config
- Interactive:
cd docker && tools/dockerBuild && tools/dockerRun
- Detached:
cd docker && tools/dockerBuild && tools/dockerRunDetached
Server GUI
https://hostname.com:443/gui
Java Client
./clientJava- Java 7+
- No dependencies
- Run test:
Python Client
./clientPython- python3
- Dependencies:
requests (pip install requests)
- Run test:
python test.py
Javascript client
./clientJavascriptclient.js is a node module- Dependencies:
packages.json -> devDependencies (npm install)
- Run test:
node test.js
- Run test in browser:
browserify test.js -o bundle.js and open test.html in browser
Certificates
- Self signed:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
- Letsencrypt: TODO
TODO
- Put data writing in a queue
- Put alert sending in a queue
- (Way) better events persister
- Pulse:
POST /{"service":"SERVICE_NAME","alertType":"ALERT_TYPE","alertTarget":"TARGET_NAME","nextIn":EXPECTED_NEXT_HEARTBEAT_IN_SECS}
- Cancel:
DELETE /{"service":"SERVICE_NAME"}
- List:
GET /
- Add:
POST /add-event{"service":"SERVICE_NAME", "level":"level", "message":"the message"}
- List:
GET /list-events- optional params:
service(is),from(above),to(below),level(above),message(contains)
./server- python3
- Dependencies:
pip install -r requirements.txt - Config:
config/config.json: main config (cp config/sample.json config/config.json)config/server.crt,config/server.key: SSL certificate, used only if SSL activated
- Datas:
data - Deploy:
- Optionnaly define pyenv-virtualenv:
pyenv virtualenv -f 3.6.8 watchTower - Install dependencies:
pip install --upgrade pip ; pip install -r requirements.txt - Setup config
- Generate ssl certificates (optional)
- Hook in with upstart (optional,
./server/config/sample.upstart.conf, http://upstart.ubuntu.com/getting-started.html)
- Optionnaly define pyenv-virtualenv:
- Run:
python server.py - Docker:
./docker- Config: place server config files in
./docker/config - Interactive:
cd docker && tools/dockerBuild && tools/dockerRun - Detached:
cd docker && tools/dockerBuild && tools/dockerRunDetached
Server GUI
https://hostname.com:443/gui
Java Client
./clientJava- Java 7+
- No dependencies
- Run test:
Python Client
./clientPython- python3
- Dependencies:
requests (pip install requests)
- Run test:
python test.py
Javascript client
./clientJavascriptclient.js is a node module- Dependencies:
packages.json -> devDependencies (npm install)
- Run test:
node test.js
- Run test in browser:
browserify test.js -o bundle.js and open test.html in browser
Certificates
- Self signed:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
- Letsencrypt: TODO
TODO
- Put data writing in a queue
- Put alert sending in a queue
- (Way) better events persister
https://hostname.com:443/gui./clientJava- Java 7+
- No dependencies
- Run test:
Python Client
./clientPython- python3
- Dependencies:
requests (pip install requests)
- Run test:
python test.py
Javascript client
./clientJavascriptclient.js is a node module- Dependencies:
packages.json -> devDependencies (npm install)
- Run test:
node test.js
- Run test in browser:
browserify test.js -o bundle.js and open test.html in browser
Certificates
- Self signed:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
- Letsencrypt: TODO
TODO
- Put data writing in a queue
- Put alert sending in a queue
- (Way) better events persister
./clientPythonrequests (pip install requests)python test.py./clientJavascriptclient.jsis a node module- Dependencies:
packages.json->devDependencies(npm install) - Run test:
node test.js - Run test in browser:
browserify test.js -o bundle.jsand opentest.htmlin browser
Certificates
- Self signed:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
- Letsencrypt: TODO
TODO
- Put data writing in a queue
- Put alert sending in a queue
- (Way) better events persister
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt- Put data writing in a queue
- Put alert sending in a queue
- (Way) better events persister
