Skip to content

Latest commit

 

History

History
51 lines (33 loc) · 2.44 KB

README.md

File metadata and controls

51 lines (33 loc) · 2.44 KB
Raw

Who watches the watcher?

This is a simple service that watches Nomad's nodes, jobs, deployments, evaluations, allocations, and task states, and writes the events to a file. The intention is that they'll be shipped to a central log collection system so users and operators can get insight into what Nomad is doing.

usage

nomad-watcher --events-file=nomad_events.json

events

The events all have the same basic structure:

  • @timestamp — Timestamp in RFC 3339 format.
  • wait_index — The value of the X-Nomad-Index header in the response to the blocking query
  • type — The type of the event; one of alloc, task_state, eval, job, node, or deploy.
  • AllocationID — The event's allocation ID, if applicable.
  • EvaluationID — The event's evaluation ID, if applicable.
  • JobID — The event's job ID, if applicable.
  • NodeID — The event's node ID, if applicable.
  • DeploymentID — The event's deployment ID, if applicable.
  • event — The details of the triggered event. This is the serialized Nomad API struct.

nomad-tail

I found myself wanting to see the events scrolling by in a console window, so I shaved a yak and created nomad-tail.

usage

nomad-tail

sample output

Everything's colorized, but you'll have to take my word for it:

<eval  > E[ddfe9d8a] system-date job-deregister pending      D[        ] N[        ] next: E[        ] prev: E[        ] block: E[        ]
<job   > system-date v001 system 050 dead
<eval  > E[ddfe9d8a] system-date job-deregister complete     D[        ] N[        ] next: E[        ] prev: E[        ] block: E[        ]
<alloc > A[b618a2de] system-date v000 E[2b880115] N[47834f73] stop alloc not needed due to job update running
<task  > A[b618a2de] system-date.date[0] dead          Killing DisplayMessage=Sent interrupt. Waiting 5s before force killing
<task  > A[b618a2de] system-date.date[0] dead          Killed DisplayMessage=Task successfully killed
<alloc > A[b618a2de] system-date v000 E[2b880115] N[47834f73] stop alloc not needed due to job update complete

No timestamps are shown to reduce clutter. iTerm2's "Show Timestamps" feature works well here.

building

You need a Go development environment. Modules are used, so Go >= 1.11 is required.

make

Binaries are placed into stage/.