forked from OpenTSDB/opentsdb.net
-
Notifications
You must be signed in to change notification settings - Fork 0
/
tcollector.html
325 lines (298 loc) · 15.7 KB
/
tcollector.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>tcollector - OpenTSDB - A Distributed, Scalable Monitoring System</title>
<link rel="stylesheet" href="css/style.css" type="text/css" media="screen"/>
<!--[if lte IE 8]>
<link rel="stylesheet" href="css/ie.css" type="text/css" media="screen"/>
<![endif]-->
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-18339382-1']);
_gaq.push(['_setDomainName', 'none']);
_gaq.push(['_setAllowLinker', true]);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<header><div id="headerbar">
<h1><a href="index.html">OpenTSDB</a></h1>
<nav><ul id="navbar">
<li><a href="overview.html">Overview</a></li>
<li><a href="getting-started.html">Getting Started</a></li>
<li><a href="manual.html">Manual</a></li>
<li><a href="faq.html">FAQ</a></li>
</ul></nav>
</div></header>
<!--[if lte IE 8]>
<div class="iesucks">Warning: You're using an unsupported, archaic browser.
Get a better, modern browsing experience with
<a href="http://www.google.com/chrome">Chrome</a> or
<a href="http://www.mozilla.com/firefox">Firefox</a>.</div>
<![endif]-->
<section id="content">
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js" type="text/javascript"></script>
<script src="misc/toc.js" type="text/javascript"></script>
<section id="tcollector">
<div id="toc"></div>
<h2>tcollector</h2>
tcollector is a client-side process that gathers data from local
collectors and pushes the data to OpenTSDB.
You run it on all your hosts, and it does the work of sending each
host's data to the TSD.<p>
OpenTSDB is designed to make it easy to collect and write data to
it. It has a simple protocol, simple enough for even a shell script
to start sending data. However, to do so reliably and consistently
is a bit harder. What do you do when your TSD server is down? How
do you make sure your collectors stay running? This is where
tcollector comes in.<p>
Tcollector does several things for you:
<ul>
<li>Runs all of your data collectors and gathers their data</li>
<li>Does all of the connection management work of sending data to the TSD</li>
<li>You don't have to embed all of this code in every collector you write</li>
<li>Does de-duplication of repeated values</li>
<li>Handles all of the wire protocol work for you, as well as future enhancements</li>
</ul>
<h3>Deduplication</h3>
Typically you want to gather data about everything in your system.
This generates a lot of datapoints, the majority of which don't
change very often over time (if ever). However, you want fine-grained
resolution when they do change. Tcollector remembers the last value
and timestamp that was sent for all of the time series for all of
the collectors it manages. If the value doesn't change between sample
intervals, it suppresses sending that datapoint. Once the value does change
(or 10 minutes have passed), it sends the last suppressed value and timestamp,
plus the current value and timestamp. In this way all of your graphs and
such are correct. Deduplication typically reduces the number of datapoints
TSD needs to collect by a large fraction. This reduces network load and
storage in the backend. A future OpenTSDB release however will improve on
the storage format by using RLE (among other things), making it essentially free
to store repeated values.
<h3>Collecting lots of metrics with tcollector</h3>
Collectors in tcollector can be written in any language. They just need to
be executable and output the data to stdout. Tcollector will handle the rest.
The collectors are placed in the <code>collectors</code> directory. Tcollector
iterates over every directory named with a number in that directory and runs all the collectors
in each directory. If you name the directory <code>60</code>, then tcollector
will try to run every collector in that directory every 60 seconds. Use
the directory <code>0</code> for any collectors that are long-lived and run
continuously. Tcollector will read their output and respawn them if they die.
Generally you want to write long-lived collectors since that has less overhead.
OpenTSDB is designed to have lots of datapoints for each metric (for most metrics
we send datapoints every 15 seconds).
<p>If there any non-numeric named directories in the <code>collectors</code> directory,
then they are ignored. We've included a <code>lib</code> and <code>etc</code> directory
for library and config data used by all collectors.
<h2>Collectors bundled with <code>tcollector</code></h2>
The following are the collectors we've included as part of the base package, together
with all of the metric names they report on and what they mean. If you have any others
you'd like to contribute, we'd love to hear about them so we can reference them or
include them with your permission in a future release.
<h3><code>0/dfstat.py</code></h3>
These stats from running <code>/usr/bin/df -PlTk</code>
<ul>
<li><code>df.1kblocks.total</code> size of fs
<li><code>df.1kblocks.used</code> 1K blocks used
<li><code>df.1kblocks.free</code> 1K blocks available
<li><code>df.inodes.total</code> total number of inodes
<li><code>df.inodes.used</code> number of inodes used
<li><code>df.inodes.free</code> number of inodes free
</ul>
These metrics include time series tagged with each mount point and the filesystem's fstype.
Since TSD doesn't allow slashes in tag values, we substitute <code>/</code> for <code>_</code>.
This collector also filters out any debugfs, devtmpfs filesystems, as well as any any
mountpoints mounted under <code>/dev/</code> or <code>/lib/</code>.
<p>With these tags you can select to graph just a specific filesystem, or all filesystems
with a particular fstype (e.g. ext3).
<h3><code>0/ifstat.py</code></h3>
These stats are from <code>/proc/net/dev</code>
<ul>
<li><code>proc.net.bytes</code> (rate) Bytes in/out</li>
<li><code>proc.net.packets</code> (rate) Packets in/out</li>
<li><code>proc.net.errs</code> (rate) Packet errors in/out</li>
<li><code>proc.net.dropped</code> (rate) Dropped packets in/out</li>
</ul>
These are interface counters, tagged with the interface,
<code>iface=</code>, and <code>direction=</code> in or out. Only
<code>eth<i>N</i></code> interfaces are tracked. We intentionally
exclude <code>bond<i>N</i></code> interfaces, because bonded
interfaces still keep counters on their child <code>eth<i>N</i></code>
interfaces and we don't want to double-count a box's network traffic
if you don't select on <code>iface=</code>.
<h3><code>0/iostat.py</code></h3>
<p>Data is from <code>/proc/diskstats</code>.
</p>
<ul>
<li><code>iostat.disk.*</code> for per-disk stats</li>
<li><code>iostat.part.*</code> for per-partition stats (see note
below on different metrics, depending on if you have a 2.6 kernel
before 2.6.25 or after.)</li>
</ul>
<p>See <a href="http://www.kernel.org/doc/Documentation/iostats.txt">iostats.txt</a>
</p><code>/proc/diskstats</code> has 11 stats for a given physical device.
These are all rate counters, except <code>ios_in_progress</code>.
</p>
<pre> .read_requests Number of reads completed
.read_merged Number of reads merged
.read_sectors Number of sectors read
.msec_read Time in msec spent reading
.write_requests Number of writes completed
.write_merged Number of writes merged
.write_sectors Number of sectors written
.msec_write Time in msec spent writing
.ios_in_progress Number of I/O operations in progress
.msec_total Time in msec doing I/O
.msec_weighted_total Weighted time doing I/O (multiplied by ios_in_progress)
</pre>
<p>in 2.6.25 and later, by-partition stats are reported the same as disks.
</p><p>NOTE: in 2.6 before 2.6.25, partitions have only 4 stats per partition
</p>
<pre> .read_issued
.read_sectors
.write_issued
.write_sectors
</pre>
<p>For partitions, these <code>*_issued</code> are counters collected before requests are merged,
so aren't the same as <code>*_requests</code> (which is post-merge, which more closely represents
represents the actual number of disk transactions).
</p><p>Given that diskstats provides both per-disk and per-partition data, for
TSDB purposes we put them under different metrics (versus the same
metric and different tags). Otherwise, if you look at a given metric, the data
for a given box will be double-counted, since a given operation will increment
both the disk series and the partition series. To fix this, we output by-disk
data to <code>iostat.disk.*</code> and by-partition data to <code>iostat.part.*</code>.
<h3><code>0/netstat.py</code></h3>
Socket allocation and network statistics.
<p>Metrics from <code>/proc/net/sockstat</code>:
<ul>
<li><code>net.sockstat.num_sockets</code>:
Number of sockets allocated (only TCP).</li>
<li><code>net.sockstat.num_timewait</code>:
Number of TCP sockets currently in <code>TIME_WAIT</code> state.</li>
<li><code>net.sockstat.sockets_inuse</code>:
Number of sockets in use (TCP/UDP/raw).</li>
<li><code>net.sockstat.num_orphans</code>:
Number of orphan TCP sockets (not attached to any file descriptor).</li>
<li><code>net.sockstat.memory</code>:
Memory allocated for this socket type (in bytes).</li>
<li><code>net.sockstat.ipfragqueues</code>:
Number of IP flows for which there are currently fragments queued for
reassembly.</li>
</ul>
<p>Metrics from <code>/proc/net/netstat</code> (<code>netstat -s</code> command):
<ul>
<li><code>net.stat.tcp.abort</code>:
Number of connections that the kernel had to abort.
<code>type=memory</code> is especially bad, the kernel had to drop a connection
due to having too many orphaned sockets. Other types are normal (e.g.
timeout).</li>
<li><code>net.stat.tcp.abort.failed</code>:
Number of times the kernel failed to abort a connection because it didn't
even have enough memory to reset it (bad).</li>
<li><code>net.stat.tcp.congestion.recovery</code>:
Number of times the kernel detected spurious retransmits and was able to
recover part or all of the CWND.</li>
<li><code>net.stat.tcp.delayedack</code>:
Number of delayed ACKs sent of different types.</li>
<li><code>net.stat.tcp.failed_accept</code>:
Number of times a connection had to be dropped after the 3WHS.
<code>reason=full_acceptq</code> indicates that the application isn't accepting
connections fast enough. You should see SYN cookies too.</li>
<li><code>net.stat.tcp.invalid_sack</code>:
Number of invalid SACKs we saw of diff types.
(requires Linux v2.6.24-rc1 or newer)</li>
<li><code>net.stat.tcp.memory.pressure</code>:
Number of times a socket entered the "memory pressure" mode (not great).</li>
<li><code>net.stat.tcp.memory.prune</code>:
Number of times a socket had to discard received data due to low memory
conditions (bad).</li>
<li><code>net.stat.tcp.packetloss.recovery</code>:
Number of times we recovered from packet loss by type of recovery (e.g.
fast retransmit vs SACK).</li>
<li><code>net.stat.tcp.receive.queue.full</code>:
Number of times a received packet had to be dropped because the socket's
receive queue was full. (requires Linux v2.6.34-rc2 or newer)</li>
<li><code>net.stat.tcp.reording</code>:
Number of times we detected re-ordering and how.</li>
<li><code>net.stat.tcp.syncookies</code>:
SYN cookies (both sent & received).</li>
</ul>
<h3><code>0/procnettcp.py</code></h3>
These stats are all from <code>/proc/net/tcp{,6}</code>. (Note if
IPv6 is enabled, some IPv4 connections seem to get put into
<code>/proc/net/tcp6</code>). Collector sleeps 60 seconds in between
intervals. Due in part to a kernel performance issue in older kernels
and in part due to systems with many TCP connections, this collector
can take sometimes 5 minutes or more to run one interval, so the
frequency of datapoints can be highly variable depending on the
system. <ul> <li><code>proc.net.tcp</code> Number of TCP connections
</ul>
<p>For each run of the collector, we classify each connection and
generate subtotals. TSD will automatically total these up when
displaying the graph, but you can drill down for each possible total
or a particular one. Each connection is broken down with a tag for
<code>user=username</code> (with a fixed list of users we care about
or put under "other" if not in the list). It is also broken down
into state with <code>state=</code>, (established, time_wait, etc). It is also
broken down into services with <code>service=</code> (http, mysql, memcache,
etc) Note that once a connection is closed, Linux seems to forget
who opened/handled the connection. For connections in time_wait,
for example, they will always show user=root. This collector does
generate a large amount of datapoints, as the number of points is
(S*(U+1)*V), where S=number of TCP states, U=Number of users you
track, and V=number of services (collections of ports). The deduper
does dedup this down very well, as only 3 of the 10 TCP states are
generally ever seen. On a typical server this can dedup down to
under 10 values per interval.
<h3><code>0/procstats.py</code></h3>
<p>Miscellaneous stats from <code>/proc</code>
</p>
<ul><li><code>proc.stat.cpu</code> (rate) CPU counters (jiffies),
tagged by cpu type (type=user, nice, system, idle, iowait, irq,
softirq, etc). As a rate they should aggregate up to approximately
100*numcpu per host. Best viewed as type=* or maybe
type={user|nice|system|iowait|irq} .
</li><li><code>proc.stat.intr</code> (rate) Number of interrupts
</li><li><code>proc.stat.ctxt</code> (rate) Number of context switches
</li></ul>
<p>See <a href="http://www.linuxhowtos.org/System/procstat.htm">http://www.linuxhowtos.org/System/procstat.htm</a>
</p>
<ul>
<li><code>proc.vmstat.*</code>A subset of VM Stats from
<code>/proc/vmstat</code> (mix of rate and non-rate) See <a
href="http://www.linuxinsight.com/proc_vmstat.html">http://www.linuxinsight.com/proc_vmstat.html</a></li>
<li><code>proc.meminfo.*</code> Memory usage stats from
<code>/proc/meminfo</code>. See the
<a href="http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/proc.txt;hb=HEAD">Linux
kernel documentation</a></li>
<li><code>proc.loadavg.*</code> 1min, 5min, 15min, runnable, total_threads
metrics from <code>/proc/loadavg</code></li>
<li><code>proc.uptime.total</code> (rate) Seconds since boot</li>
<li><code>proc.uptime.now</code> (rate) Seconds since boot that the system has been idle</li>
<li><code>proc.kernel.entropy_avail</code>: Amount of entropy (in bits)
available in the input pool (the one that's cryptographically strong and
backing <code>/dev/random</code> among other things). Watch this value on
your frontend servers that do SSL unwrapping, if it gets too low, your SSL
performance will suffer.</li>
<li><code>sys.numa.zoneallocs</code>: Number of pages allocated from the
preferred node (<code>type=hit</code>) or not (<code>type=miss</code>).</li>
<li><code>sys.numa.foreign_allocs</code>: Number of pages this node allocated
because the preferred node didn't have a free page to accommodate the request.</li>
<li><code>sys.numa.allocation</code>: Number of pages allocated locally
(<code>type=local</code>) or remotely (<code>type=remote</code>) for processes
executing on this node.</li>
<li><code>sys.numa.interleave</code>: Number of pages allocated successfully
by the interleave strategy.</li>
</ul>
</section>
<footer>
© 2010–2012 The OpenTSDB Authors.
</footer>
</section></body></html>