423 lines
12 KiB
Markdown
423 lines
12 KiB
Markdown
# [Watchdog](https://git.rootprojects.org/root/watchdog.go)
|
||
|
||
> Get notified when sites go down.
|
||
|
||
Watchdog: Webhooks for all the times when something doesn't go right.
|
||
|
||
For every `url` listed in the config, check if the specified `keywords` are found on the page every 5 minutes.
|
||
Whenever they aren't found execute the `recover_script` and make a request to each of the associated `webhooks`.
|
||
|
||
Can work with email, text (sms), push notifications, etc.
|
||
|
||
# Install
|
||
|
||
## Downloads
|
||
|
||
### MacOS
|
||
|
||
MacOS (darwin): [64-bit Download ](https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog -o watchdog
|
||
```
|
||
|
||
### Windows
|
||
|
||
<details>
|
||
<summary>See download options</summary>
|
||
Windows 10: [64-bit Download](https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe)
|
||
|
||
```
|
||
powershell.exe $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe -OutFile watchdog.exe
|
||
```
|
||
|
||
Windows 7: [32-bit Download](https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe)
|
||
|
||
```
|
||
powershell.exe "(New-Object Net.WebClient).DownloadFile('https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe', 'watchdog.exe')"
|
||
```
|
||
|
||
</details>
|
||
|
||
### Linux
|
||
|
||
<details>
|
||
<summary>See download options</summary>
|
||
|
||
Linux (64-bit): [Download](https://rootprojects.org/watchdog/dist/linux/amd64/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/linux/amd64/watchdog -o watchdog
|
||
```
|
||
|
||
Linux (32-bit): [Download](https://rootprojects.org/watchdog/dist/linux/386/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/linux/386/watchdog -o watchdog
|
||
```
|
||
|
||
</details>
|
||
|
||
### Raspberry Pi (Linux ARM)
|
||
|
||
<details>
|
||
<summary>See download options</summary>
|
||
|
||
RPi 4 (64-bit armv8): [Download](https://rootprojects.org/watchdog/dist/linux/armv8/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/linux/armv8/watchdog -o watchdog`
|
||
```
|
||
|
||
RPi 3 (armv7): [Download](https://rootprojects.org/watchdog/dist/linux/armv7/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/linux/armv7/watchdog -o watchdog
|
||
```
|
||
|
||
ARMv6: [Download](https://rootprojects.org/watchdog/dist/linux/armv6/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/linux/armv6/watchdog -o watchdog
|
||
```
|
||
|
||
RPi Zero (armv5): [Download](https://rootprojects.org/watchdog/dist/linux/armv5/watchdog)
|
||
|
||
```
|
||
curl https://rootprojects.org/watchdog/dist/linux/armv5/watchdog -o watchdog
|
||
```
|
||
|
||
</details>
|
||
|
||
## Git:
|
||
|
||
```bash
|
||
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
|
||
pushd watchdog.go/
|
||
go generate -mod=vendor ./...
|
||
pushd cmd/watchdog
|
||
go build -mod=vendor
|
||
```
|
||
|
||
# Usage
|
||
|
||
Mac, Linux:
|
||
|
||
```bash
|
||
./watchdog -c config.json
|
||
```
|
||
|
||
Windows:
|
||
|
||
```bash
|
||
watchdog.exe -c config.json
|
||
```
|
||
|
||
# Changelog
|
||
|
||
- v1.2.0
|
||
- report when sites come back up
|
||
- and more template vars
|
||
- and localization for status
|
||
- v1.1.0 support `json` request bodies (for Pushbullet)
|
||
- v1.0.0 support Twilio and Mailgun
|
||
|
||
# Getting Started
|
||
|
||
<details>
|
||
<summary>How do I configure what to watch?</summary>
|
||
|
||
### `name`
|
||
|
||
This is an arbitrary name which can be used as `{{ .Name }}` in the template strings.
|
||
|
||
In most cases I would probably set mine to the literal domain name (such as `git.rootprojects.org`),
|
||
but you can do whatever makes sense to you.
|
||
|
||
If you use it in a template, you probably shouldn't allow it to be arbitrary user input.
|
||
|
||
### `url`
|
||
|
||
This is the page the watchdog will check for the exact match of `keywords` you set.
|
||
|
||
It's a get request. I actually want to be regularly check that contact forms are working,
|
||
so I may very well add more functionality to this in the future.
|
||
|
||
### `keywords`
|
||
|
||
The `url` will be checked for a literal, exact match of keywords.
|
||
|
||
Be careful of "smart quotes" and HTML entities:
|
||
|
||
- `We’re Open!` is not `We're Open!`
|
||
- Neither is `We're Open!` nor `We're Open!`
|
||
|
||
Leave empty for No Content pages, such as redirects.
|
||
|
||
### `badwords`
|
||
|
||
The opposite of `keywords`.
|
||
|
||
If a literal, exact match of badwords exists as part of the response, the site is considered to be down.
|
||
|
||
Ignored if empty.
|
||
|
||
### `localizations`
|
||
|
||
Normally `{{ .Status }}` will be `"up"` or `"down"` and `{{ .Message }}` will be `"is down"` or `"came back up"`.
|
||
Localizations allow you to swap that out for something else.
|
||
|
||
I added this so that I could use "🔥🔥🔥" and "👍" for myself without imposing upon others.
|
||
|
||
### `webhooks`
|
||
|
||
This references the arbitrary `name` of a webhook in the `webhooks` array.
|
||
|
||
These webhooks will be run in order and the next will still be run even when the previous fails.
|
||
|
||
### `recover_script`
|
||
|
||
The full contents of this file will be passed to bash as if it were a file.
|
||
|
||
You can run a single line, such as `ssh watchdog@my.example.com 'systemctl restart foo.service'`
|
||
or an entire script.
|
||
|
||
#### Pro-Tip™
|
||
|
||
Don't forget that you can add single-use ssh keys to run specific commands on a remote system.
|
||
|
||
On the system that runs the watchdog you would create a new, single-use key, like this:
|
||
|
||
```bash
|
||
ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
|
||
cat ~/.ssh/restart-service-foo.pub
|
||
```
|
||
|
||
On the system that runs the service being watched you would add that key restricted to the single command:
|
||
|
||
`/root/.ssh/authorized_keys`:
|
||
|
||
```
|
||
command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog
|
||
```
|
||
|
||
</details>
|
||
<details>
|
||
<summary>{{ .Name }} and other template variables</summary>
|
||
|
||
- `{{ .Name }}` is the name of your site.
|
||
- `{{ .Message }}` is either `went down` or `came back up`.
|
||
- `{{ .Status }}` is either `up` or `down`.
|
||
- `{{ .Watchdog }}` is the name of your watchdog (useful if you have multiple).
|
||
|
||
It refers to the name of the watch, which is "Example Site" in the sample config below.
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Mailgun</summary>
|
||
|
||
### `my_mailgun`
|
||
|
||
Replace `my_mailgun` with whatever name you like, or leave it the same.
|
||
|
||
It's an arbitrary name for you to reference.
|
||
For example, you may have different mailgun configurations for different domains.
|
||
|
||
### `my.example.com`
|
||
|
||
Replace both instances of `my.example.com` with your domain in mailgun.
|
||
|
||
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
|
||
Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
with your API token.
|
||
|
||
`api` stays, it is the expected "username".
|
||
|
||
### `to`, `from`, `subject`, & `text`
|
||
|
||
Hopefully it's obivous enough that you should send your alerts to yourself,
|
||
not `jon.doe@gmail.com` but, just in case: change that to your email.
|
||
|
||
The `from` address can be _any_ domain in your mailgun account.
|
||
|
||
`subject` is the plain-text subject and `text` must be plain-text (not html) email contents.
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Pushbullet</summary>
|
||
|
||
Pushbullet is a push notification service that I found pretty easy to work with.
|
||
I logged in with my iPhone through facebook, as well as in a web browser,
|
||
grabbed the API token, and I was able to test the documentation all in just a few minutes.
|
||
|
||
### `my_pushbullet`
|
||
|
||
Replace `my_pushbullet` with whatever name you like, or leave it the same.
|
||
|
||
It's an arbitrary name for you to reference.
|
||
For example, you may have different Pushbullet configurations for different domains.
|
||
|
||
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
|
||
Replace `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in the HTTP Access-Token header
|
||
with your API token.
|
||
|
||
You'll find this on the Pushbullet dashboard under Settings, Account, Access Tokens,
|
||
and you'll have to click to create it:
|
||
|
||
- <https://www.pushbullet.com/#settings/account>
|
||
|
||
The example was taken from the Pushbullet API documentation:
|
||
|
||
- <https://docs.pushbullet.com/#create-push>
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Twilio</summary>
|
||
|
||
### `my_twilio`
|
||
|
||
Replace `my_twilio` with whatever name you like, or leave it the same.
|
||
|
||
It's an arbitrary name for you to reference.
|
||
For example, you may have different twilio configurations for different domains.
|
||
|
||
### `AC00000000000000000000000000000000`
|
||
|
||
This is a placeholder for your `sid`, also called "Account SID".
|
||
|
||
Replace both instances of `AC00000000000000000000000000000000` with the `sid` shown in your twilio dashboard.
|
||
|
||
This is used both in the `url` as well as in the `auth`.
|
||
|
||
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
|
||
Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
with your API token.
|
||
|
||
You'll find this on the twilio dashboard with your "Account SID",
|
||
but you'll have to click to reveal it.
|
||
|
||
### `To`, `From`, & `Body`
|
||
|
||
All phone numbers should have the country code prefix (`+1` for USA) attached.
|
||
|
||
`To` is the number that you want to send the text to.
|
||
|
||
`From` must be a number listed in your twilio account.
|
||
|
||
`Body` is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Nexmo, Mailjet, and other webhook-enabled services</summary>
|
||
|
||
See the examples of Mailgun, Pushbullet, Twilio.
|
||
|
||
Look for "curl" in the documentation of the service that you're using.
|
||
It should be fairly easy to just look at the headers that are being set and repeat.
|
||
|
||
</details>
|
||
|
||
## Sample Config
|
||
|
||
You can set notifications for _any_ service that supports HTTPS webhooks.
|
||
|
||
The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from their `curl` documentation.
|
||
|
||
```json
|
||
{
|
||
"watchdog": "Monitor A",
|
||
"watches": [
|
||
{
|
||
"name": "Example Site",
|
||
"url": "https://example.com/",
|
||
"keywords": "My Site",
|
||
"badwords": "Could not connect to database.",
|
||
"webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
|
||
"recover_script": "systemctl restart example-site"
|
||
}
|
||
],
|
||
"webhooks": [
|
||
{
|
||
"name": "my_mailgun",
|
||
"method": "POST",
|
||
"url": "https://api.mailgun.net/v3/my.example.com/messages",
|
||
"auth": {
|
||
"username": "api",
|
||
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||
},
|
||
"headers": {
|
||
"User-Agent": "Watchdog/1.0"
|
||
},
|
||
"form": {
|
||
"from": "Watchdog <watchdog@my.example.com>",
|
||
"to": "jon.doe@gmail.com",
|
||
"subject": "[{{ .Watchdog }}] {{ .Name }} {{ .Message }}.",
|
||
"text": "{{ .Name }} {{ .Message }}. Reported by {{ .Watchdog }}."
|
||
}
|
||
},
|
||
{
|
||
"name": "my_pushbullet",
|
||
"method": "POST",
|
||
"url": "https://api.pushbullet.com/v2/pushes",
|
||
"headers": {
|
||
"Access-Token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
|
||
"User-Agent": "Watchdog/1.0"
|
||
},
|
||
"json": {
|
||
"body": "The system {{ .Message }}. Check up on {{ .Name }} ASAP.",
|
||
"title": "{{ .Name }} {{ .Message }}.",
|
||
"type": "note"
|
||
}
|
||
},
|
||
{
|
||
"name": "my_twilio",
|
||
"method": "POST",
|
||
"url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
|
||
"auth": {
|
||
"username": "AC00000000000000000000000000000000",
|
||
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||
},
|
||
"headers": {
|
||
"User-Agent": "Watchdog/1.0"
|
||
},
|
||
"form": {
|
||
"To": "+1 801 555 1234",
|
||
"From": "+1 800 555 4321",
|
||
"Body": "[{{ .Name }}] The system is down. The system is down."
|
||
}
|
||
}
|
||
],
|
||
"localizations": {
|
||
"up": "👍",
|
||
"down": "🔥🔥🔥"
|
||
}
|
||
}
|
||
```
|
||
|
||
<!--
|
||
I may want to allow for an overlay of configs for cases like Twilio
|
||
where it must be run once per contact.
|
||
|
||
"webhooks": [
|
||
{
|
||
"name": "twilio",
|
||
"config": {
|
||
"sid": "AC00000000000000000000000000000000",
|
||
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||
},
|
||
"configs": [{ "To": "+1 555 555 5555", "From": "+1 555 234 5678" }],
|
||
"url": "https://api.twilio.com/2010-04-01/Accounts/{{ .sid }}/Messages.json",
|
||
"auth": { "user": "{{ .sid }}", "pass": "{{ .token }}" },
|
||
"_TODO_Body": "switch template if hardfail",
|
||
"form": {
|
||
"To": "{{ .To }}",
|
||
"From": "{{ .From }}",
|
||
"Body": "[{{ .name }}] The system is down. The system is down."
|
||
}
|
||
}
|
||
]
|
||
-->
|