go-watchdog/README.md

# [Watchdog](https://git.rootprojects.org/root/watchdog.go)

> Get notified when sites go down.

Watchdog: Webhooks for all the times when something doesn't go right.

For every `url` listed in the config, check if the specified `keywords` are found on the page every 5 minutes.
Whenever they aren't found execute the `recover_script` and make a request to each of the associated `webhooks`.

Can work with email, text (sms), push notifications, etc.

# Install

## Downloads

### MacOS

MacOS (darwin): [64-bit Download ](https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog)

```
curl https://rootprojects.org/watchdog/dist/darwin/amd64/watchdog -o watchdog
```

### Windows

<details>
<summary>See download options</summary>
Windows 10: [64-bit Download](https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe)

```
powershell.exe $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest https://rootprojects.org/watchdog/dist/windows/amd64/watchdog.exe -OutFile watchdog.exe
```

Windows 7: [32-bit Download](https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe)

```
powershell.exe "(New-Object Net.WebClient).DownloadFile('https://rootprojects.org/watchdog/dist/windows/386/watchdog.exe', 'watchdog.exe')"
```

</details>

### Linux

<details>
<summary>See download options</summary>

Linux (64-bit): [Download](https://rootprojects.org/watchdog/dist/linux/amd64/watchdog)

```
curl https://rootprojects.org/watchdog/dist/linux/amd64/watchdog -o watchdog
```

Linux (32-bit): [Download](https://rootprojects.org/watchdog/dist/linux/386/watchdog)

```
curl https://rootprojects.org/watchdog/dist/linux/386/watchdog -o watchdog
```

</details>

### Raspberry Pi (Linux ARM)

<details>
<summary>See download options</summary>

RPi 4 (64-bit armv8): [Download](https://rootprojects.org/watchdog/dist/linux/armv8/watchdog)

```
curl https://rootprojects.org/watchdog/dist/linux/armv8/watchdog -o watchdog`
```

RPi 3 (armv7): [Download](https://rootprojects.org/watchdog/dist/linux/armv7/watchdog)

```
curl https://rootprojects.org/watchdog/dist/linux/armv7/watchdog -o watchdog
```

ARMv6: [Download](https://rootprojects.org/watchdog/dist/linux/armv6/watchdog)

```
curl https://rootprojects.org/watchdog/dist/linux/armv6/watchdog -o watchdog
```

RPi Zero (armv5): [Download](https://rootprojects.org/watchdog/dist/linux/armv5/watchdog)

```
curl https://rootprojects.org/watchdog/dist/linux/armv5/watchdog -o watchdog
```

</details>

## Git:

```bash
git clone https://git.coolaj86.com/coolaj86/watchdog.go.git
pushd watchdog.go/
go generate -mod=vendor ./...
pushd cmd/watchdog
go build -mod=vendor
```

# Usage

Mac, Linux:

```bash
./watchdog -c config.json
```

Windows:

```bash
watchdog.exe -c config.json
```

# Changelog

- v1.2.0
  - report when sites come back up
  - and more template vars
  - and localization for status
- v1.1.0 support `json` request bodies (for Pushbullet)
- v1.0.0 support Twilio and Mailgun

# Getting Started

<details>
  <summary>How do I configure what to watch?</summary>

### `name`

This is an arbitrary name which can be used as `{{ .Name }}` in the template strings.

In most cases I would probably set mine to the literal domain name (such as `git.rootprojects.org`),
but you can do whatever makes sense to you.

If you use it in a template, you probably shouldn't allow it to be arbitrary user input.

### `url`

This is the page the watchdog will check for the exact match of `keywords` you set.

It's a get request. I actually want to be regularly check that contact forms are working,
so I may very well add more functionality to this in the future.

### `keywords`

The `url` will be checked for a literal, exact match of keywords.

Be careful of "smart quotes" and HTML entities:

- `We’re Open!` is not `We're Open!`
- Neither is `We&apos;re Open!` nor `We&#39;re Open!`

Leave empty for No Content pages, such as redirects.

### `badwords`

The opposite of `keywords`.

If a literal, exact match of badwords exists as part of the response, the site is considered to be down.

Ignored if empty.

### `localizations`

Normally `{{ .Status }}` will be `"up"` or `"down"` and `{{ .Message }}` will be `"is down"` or `"came back up"`.
Localizations allow you to swap that out for something else.

I added this so that I could use "🔥🔥🔥" and "👍" for myself without imposing upon others.

### `webhooks`

This references the arbitrary `name` of a webhook in the `webhooks` array.

These webhooks will be run in order and the next will still be run even when the previous fails.

### `recover_script`

The full contents of this file will be passed to bash as if it were a file.

You can run a single line, such as `ssh watchdog@my.example.com 'systemctl restart foo.service'`
or an entire script.

#### Pro-Tip™

Don't forget that you can add single-use ssh keys to run specific commands on a remote system.

On the system that runs the watchdog you would create a new, single-use key, like this:

```bash
ssh-keygen -N '' -f ~/.ssh/restart-service-foo -C "For Foo's Watchdog"
cat ~/.ssh/restart-service-foo.pub
```

On the system that runs the service being watched you would add that key restricted to the single command:

`/root/.ssh/authorized_keys`:

```
command="systemctl restart foo.service",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5HKrWMTkrInDNGMeEWu/bSc2RCCpUPBbZCh1hWIASzMUlPDdTtqwe6ve4aozbeSVFm0mTfBlNPWPqf1ZAx80lQTbYIPNsPlQw0ktuFPWHqDGayQuPBwtk7PlzOcRah29EZ/gbz4vQCDk5G1AygBBt9S3US7Q+/xi5mk/bKkmenpYUwBrpaHx/hm4xY/6qThZDh+tf5CvTnbnb3tVS8ldOIOHMMBtxkzOcgxu12gJV3cHg/xvNd1fTrcshnjnldQphhW0/g068Ibz6aabjuy5h89uVvbEv74wV5CH7XS0TsuOIhv9/dKoi9XBRI9oM4RgPNLWxZETOGzKGYOHqxcmL For Foo's Watchdog
```

</details>
<details>
  <summary>{{ .Name }} and other template variables</summary>

- `{{ .Name }}` is the name of your site.
- `{{ .Message }}` is either `went down` or `came back up`.
- `{{ .Status }}` is either `up` or `down`.
- `{{ .Watchdog }}` is the name of your watchdog (useful if you have multiple).

It refers to the name of the watch, which is "Example Site" in the sample config below.

</details>
<details>
  <summary>How to use with Mailgun</summary>

### `my_mailgun`

Replace `my_mailgun` with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference.
For example, you may have different mailgun configurations for different domains.

### `my.example.com`

Replace both instances of `my.example.com` with your domain in mailgun.

### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`

Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
with your API token.

`api` stays, it is the expected "username".

### `to`, `from`, `subject`, &amp; `text`

Hopefully it's obivous enough that you should send your alerts to yourself,
not `jon.doe@gmail.com` but, just in case: change that to your email.

The `from` address can be _any_ domain in your mailgun account.

`subject` is the plain-text subject and `text` must be plain-text (not html) email contents.

</details>
<details>
  <summary>How to use with Pushbullet</summary>

Pushbullet is a push notification service that I found pretty easy to work with.
I logged in with my iPhone through facebook, as well as in a web browser,
grabbed the API token, and I was able to test the documentation all in just a few minutes.

### `my_pushbullet`

Replace `my_pushbullet` with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference.
For example, you may have different Pushbullet configurations for different domains.

### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`

Replace `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in the HTTP Access-Token header
with your API token.

You'll find this on the Pushbullet dashboard under Settings, Account, Access Tokens,
and you'll have to click to create it:

- <https://www.pushbullet.com/#settings/account>

The example was taken from the Pushbullet API documentation:

- <https://docs.pushbullet.com/#create-push>

</details>
<details>
  <summary>How to use with Twilio</summary>

### `my_twilio`

Replace `my_twilio` with whatever name you like, or leave it the same.

It's an arbitrary name for you to reference.
For example, you may have different twilio configurations for different domains.

### `AC00000000000000000000000000000000`

This is a placeholder for your `sid`, also called "Account SID".

Replace both instances of `AC00000000000000000000000000000000` with the `sid` shown in your twilio dashboard.

This is used both in the `url` as well as in the `auth`.

### `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`

Replace the HTTP Basic Auth "password" `xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx`
with your API token.

You'll find this on the twilio dashboard with your "Account SID",
but you'll have to click to reveal it.

### `To`, `From`, &amp; `Body`

All phone numbers should have the country code prefix (`+1` for USA) attached.

`To` is the number that you want to send the text to.

`From` must be a number listed in your twilio account.

`Body` is a plain-text short message. Remember K.I.S.S: "keep it short 'n sweet".

</details>
<details>
  <summary>How to use with Nexmo, Mailjet, and other webhook-enabled services</summary>

See the examples of Mailgun, Pushbullet, Twilio.

Look for "curl" in the documentation of the service that you're using.
It should be fairly easy to just look at the headers that are being set and repeat.

</details>

## Sample Config

You can set notifications for _any_ service that supports HTTPS webhooks.

The examples below are shown with Mailgun, Pushbullet, and Twilio, as taken from their `curl` documentation.

```json
{
  "watchdog": "Monitor A",
  "watches": [
    {
      "name": "Example Site",
      "url": "https://example.com/",
      "keywords": "My Site",
      "badwords": "Could not connect to database.",
      "webhooks": ["my_mailgun", "my_pushbullet", "my_twilio"],
      "recover_script": "systemctl restart example-site"
    }
  ],
  "webhooks": [
    {
      "name": "my_mailgun",
      "method": "POST",
      "url": "https://api.mailgun.net/v3/my.example.com/messages",
      "auth": {
        "username": "api",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "headers": {
        "User-Agent": "Watchdog/1.0"
      },
      "form": {
        "from": "Watchdog <watchdog@my.example.com>",
        "to": "jon.doe@gmail.com",
        "subject": "[{{ .Watchdog }}] {{ .Name }} {{ .Message }}.",
        "text": "{{ .Name }} {{ .Message }}. Reported by {{ .Watchdog }}."
      }
    },
    {
      "name": "my_pushbullet",
      "method": "POST",
      "url": "https://api.pushbullet.com/v2/pushes",
      "headers": {
        "Access-Token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "User-Agent": "Watchdog/1.0"
      },
      "json": {
        "body": "The system {{ .Message }}. Check up on {{ .Name }} ASAP.",
        "title": "{{ .Name }} {{ .Message }}.",
        "type": "note"
      }
    },
    {
      "name": "my_twilio",
      "method": "POST",
      "url": "https://api.twilio.com/2010-04-01/Accounts/AC00000000000000000000000000000000/Messages.json",
      "auth": {
        "username": "AC00000000000000000000000000000000",
        "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "headers": {
        "User-Agent": "Watchdog/1.0"
      },
      "form": {
        "To": "+1 801 555 1234",
        "From": "+1 800 555 4321",
        "Body": "[{{ .Name }}] The system is down. The system is down."
      }
    }
  ],
  "localizations": {
    "up": "👍",
    "down": "🔥🔥🔥"
  }
}
```

<!--
I may want to allow for an overlay of configs for cases like Twilio
where it must be run once per contact.

  "webhooks": [
    {
      "name": "twilio",
      "config": {
        "sid": "AC00000000000000000000000000000000",
        "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      },
      "configs": [{ "To": "+1 555 555 5555", "From": "+1 555 234 5678" }],
      "url": "https://api.twilio.com/2010-04-01/Accounts/{{ .sid }}/Messages.json",
      "auth": { "user": "{{ .sid }}", "pass": "{{ .token }}" },
      "_TODO_Body": "switch template if hardfail",
      "form": {
        "To": "{{ .To }}",
        "From": "{{ .From }}",
        "Body": "[{{ .name }}] The system is down. The system is down."
      }
    }
  ]
-->