323 lines
10 KiB
Markdown
323 lines
10 KiB
Markdown
# [Watchdog](https://git.rootprojects.org/root/watchdog.go)
|
||
|
||
> Get notified when sites go down.
|
||
|
||
Watchdog: Webhooks for all the times when something doesn't go right.
|
||
|
||
For every `url` listed in the config, check if the specified `keywords` are found on the page every 5 minutes.
|
||
Whenever they aren't found execute the `recover_script` and make a request to each of the associated `webhooks`.
|
||
|
||
Can work with email, text (sms), push notifications, etc.
|
||
|
||
# Install
|
||
|
||
Git:
|
||
|
||
```bash
|
||
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
|
||
pushd watchdog.go/
|
||
go generate -mod=vendor ./...
|
||
pushd cmd/watchdog
|
||
go build -mod=vendor
|
||
```
|
||
|
||
Zip:
|
||
|
||
- Linux
|
||
- [watchdog-v1.1.0-linux-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-amd64.zip)
|
||
- [watchdog-v1.1.0-linux-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-386.zip)
|
||
- [watchdog-v1.1.0-linux-armv7.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv7.zip)
|
||
- [watchdog-v1.1.0-linux-armv5.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-linux/watchdog-v1.1.0-linux-armv5.zip)
|
||
- MacOS
|
||
- [watchdog-v1.1.0-darwin-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-macos/watchdog-v1.1.0-darwin-amd64.zip)
|
||
- Windows
|
||
- [watchdog-v1.1.0-windows-amd64.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-amd64.zip)
|
||
- [watchdog-v1.1.0-windows-386.zip](https://git.rootprojects.org/root/watchdog.go/releases/download/v1.1.0-windows/watchdog-v1.1.0-windows-386.zip)
|
||
|
||
# Usage
|
||
|
||
Mac, Linux:
|
||
|
||
```bash
|
||
./watchdog -c config.json
|
||
```
|
||
|
||
Windows:
|
||
|
||
```bash
|
||
watchdog.exe -c config.json
|
||
```
|
||
|
||
# Getting Started
|
||
|
||
<details>
|
||
<summary>How do I configure what to watch?</summary>
|
||
|
||
### `name`
|
||
|
||
This is an arbitrary name which can be used as `{{ .Name }}` in the template strings.
|
||
|
||
In most cases I would probably set mine to the literal domain name (such as `git.rootprojects.org`),
|
||
but you can do whatever makes sense to you.
|
||
|
||
If you use it in a template, you probably shouldn't allow it to be arbitrary user input.
|
||
|
||
### `url`
|
||
|
||
This is the page the watchdog will check for the exact match of `keywords` you set.
|
||
|
||
It's a get request. I actually want to be regularly check that contact forms are working,
|
||
so I may very well add more functionality to this in the future.
|
||
|
||
### `keywords`
|
||
|
||
The `url` will be checked for a literal, exact match of keywords.
|
||
|
||
Be careful of "smart quotes" and HTML entities:
|
||
|
||
- `We’re Open!` is not `We're Open!`
|
||
- Neither is `We're Open!` nor `We're Open!`
|
||
|
||
### `webhooks`
|
||
|
||
This references the arbitrary `name` of a webhook in the `webhooks` array.
|
||
|
||
These webhooks will be run in order and the next will still be run even when the previous fails.
|
||
|
||
### `recover_script`
|
||
|
||
The full contents of this file will be passed to bash as if it were a file.
|
||
|
||
You can run a single line, such as `ssh watchdog@my.example.com 'systemctl restart foo.service'`
|
||
or an entire script.
|
||
|
||
#### Pro-Tip™
|
||
|
||
Don't forget that you can add single-use ssh keys to run specific commands on a remote system.
|
||
|
||
On the system that runs the watchdog you would create a new, single-use key, like this:
|
||
|
||
```bash
|
||
ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
|
||
cat ~/.ssh/restart-service-foo.pub
|
||
```
|
||
|
||
On the system that runs the service being watched you would add that key restricted to the single command:
|
||
|
||
`/root/.ssh/authorized_keys`:
|
||
|
||
```
|
||
command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog
|
||
```
|
||
|
||
</details>
|
||
<details>
|
||
<summary>{{ .Name }} and other template variables</summary>
|
||
|
||
`{{ .Name }}` is the only template variable right now.
|
||
|
||
It refers to the name of the watch, which is "Example Site" in the sample config below.
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Mailgun</summary>
|
||
|
||
### `my_mailgun`
|
||
|
||
Replace `my_mailgun` with whatever name you like, or leave it the same.
|
||
|
||
It's an arbitrary name for you to reference.
|
||
For example, you may have different mailgun configurations for different domains.
|
||
|
||
### `my.example.com`
|
||
|
||
Replace both instances of `my.example.com` with your domain in mailgun.
|
||
|
||
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
|
||
Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
with your API token.
|
||
|
||
`api` stays, it is the expected "username".
|
||
|
||
### `to`, `from`, `subject`, & `text`
|
||
|
||
Hopefully it's obivous enough that you should send your alerts to yourself,
|
||
not `jon.doe@gmail.com` but, just in case: change that to your email.
|
||
|
||
The `from` address can be _any_ domain in your mailgun account.
|
||
|
||
`subject` is the plain-text subject and `text` must be plain-text (not html) email contents.
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Pushbullet</summary>
|
||
|
||
Pushbullet is a push notification service that I found pretty easy to work with.
|
||
I logged in with my iPhone through facebook, as well as in a web browser,
|
||
grabbed the API token, and I was able to test the documentation all in just a few minutes.
|
||
|
||
### `my_pushbullet`
|
||
|
||
Replace `my_pushbullet` with whatever name you like, or leave it the same.
|
||
|
||
It's an arbitrary name for you to reference.
|
||
For example, you may have different Pushbullet configurations for different domains.
|
||
|
||
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
|
||
Replace `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in the HTTP Access-Token header
|
||
with your API token.
|
||
|
||
You'll find this on the Pushbullet dashboard under Settings, Account, Access Tokens,
|
||
and you'll have to click to create it:
|
||
|
||
- <https://www.pushbullet.com/#settings/account>
|
||
|
||
The example was taken from the Pushbullet API documentation:
|
||
|
||
- <https://docs.pushbullet.com/#create-push>
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Twilio</summary>
|
||
|
||
### `my_twilio`
|
||
|
||
Replace `my_twilio` with whatever name you like, or leave it the same.
|
||
|
||
It's an arbitrary name for you to reference.
|
||
For example, you may have different twilio configurations for different domains.
|
||
|
||
### `AC00000000000000000000000000000000`
|
||
|
||
This is a placeholder for your `sid`, also called "Account SID".
|
||
|
||
Replace both instances of `AC00000000000000000000000000000000` with the `sid` shown in your twilio dashboard.
|
||
|
||
This is used both in the `url` as well as in the `auth`.
|
||
|
||
### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
|
||
Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
|
||
with your API token.
|
||
|
||
You'll find this on the twilio dashboard with your "Account SID",
|
||
but you'll have to click to reveal it.
|
||
|
||
### `To`, `From`, & `Body`
|
||
|
||
All phone numbers should have the country code prefix (`+1` for USA) attached.
|
||
|
||
`To` is the number that you want to send the text to.
|
||
|
||
`From` must be a number listed in your twilio account.
|
||
|
||
`Body` is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".
|
||
|
||
</details>
|
||
<details>
|
||
<summary>How to use with Nexmo, Mailjet, and other webhook-enabled services</summary>
|
||
|
||
See the examples of Mailgun, Pushbullet, Twilio.
|
||
|
||
Look for "curl" in the documentation of the service that you're using.
|
||
It should be fairly easy to just look at the headers that are being set and repeat.
|
||
|
||
</details>
|
||
|
||
## Sample Config
|
||
|
||
You can set notifications for _any_ service that supports HTTPS webhooks.
|
||
|
||
The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from their `curl` documentation.
|
||
|
||
```json
|
||
{
|
||
"watches": [
|
||
{
|
||
"name": "Example Site",
|
||
"url": "https://example.com/",
|
||
"keywords": "My Site",
|
||
"webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
|
||
"recover_script": "systemctl restart example-site"
|
||
}
|
||
],
|
||
"webhooks": [
|
||
{
|
||
"name": "my_mailgun",
|
||
"method": "POST",
|
||
"url": "https://api.mailgun.net/v3/my.example.com/messages",
|
||
"auth": {
|
||
"username": "api",
|
||
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||
},
|
||
"headers": {
|
||
"User-Agent": "Watchdog/1.0"
|
||
},
|
||
"form": {
|
||
"from": "Watchdog <watchdog@my.example.com>",
|
||
"to": "jon.doe@gmail.com",
|
||
"subject": "{{ .Name }} is down.",
|
||
"text": "The system is down. Check up on {{ .Name }} ASAP."
|
||
}
|
||
},
|
||
{
|
||
"name": "my_pushbullet",
|
||
"method": "POST",
|
||
"url": "https://api.pushbullet.com/v2/pushes",
|
||
"headers": {
|
||
"Access-Token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
|
||
"User-Agent": "Watchdog/1.0"
|
||
},
|
||
"json": {
|
||
"body": "The system is down. Check up on {{ .Name }} ASAP.",
|
||
"title": "{{ .Name }} is down.",
|
||
"type": "note"
|
||
}
|
||
},
|
||
{
|
||
"name": "my_twilio",
|
||
"method": "POST",
|
||
"url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
|
||
"auth": {
|
||
"username": "AC00000000000000000000000000000000",
|
||
"password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||
},
|
||
"headers": {
|
||
"User-Agent": "Watchdog/1.0"
|
||
},
|
||
"form": {
|
||
"To": "+1 801 555 1234",
|
||
"From": "+1 800 555 4321",
|
||
"Body": "[{{ .Name }}] The system is down. The system is down."
|
||
}
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
<!--
|
||
I may want to allow for an overlay of configs for cases like Twilio
|
||
where it must be run once per contact.
|
||
|
||
"webhooks": [
|
||
{
|
||
"name": "twilio",
|
||
"config": {
|
||
"sid": "AC00000000000000000000000000000000",
|
||
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
|
||
},
|
||
"configs": [{ "To": "+1 555 555 5555", "From": "+1 555 234 5678" }],
|
||
"url": "https://api.twilio.com/2010-04-01/Accounts/{{ .sid }}/Messages.json",
|
||
"auth": { "user": "{{ .sid }}", "pass": "{{ .token }}" },
|
||
"_TODO_Body": "switch template if hardfail",
|
||
"form": {
|
||
"To": "{{ .To }}",
|
||
"From": "{{ .From }}",
|
||
"Body": "[{{ .name }}] The system is down. The system is down."
|
||
}
|
||
}
|
||
]
|
||
-->
|