Monitoring File System Latency with fsstat

Solaris 11.4 (beta) comes with a new version of the fsstat command. Why is this interesting? Because with the newly added -l option now fsstat is able to repot latency information on read, write, and readdir operations for file system types or individual file systems.
The latency information is independent of physical I/O operations
Therefore it is representative of file system performance, as seen by applications which helps improving troubleshooting performance issues in that area.

One small step for fsstat, one giant leap for troubleshooting enhancement and user experience.

Here is a quick example:

root@wacken:~# fsstat zfs proc tmpfs 1
 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
71.0K 44.7K 7.51K 1.42M 1.22K  6.93M 26.8K  240K 1019M  111K 3.61G zfs
    0     0     0 8.95K     0  19.8K 63.0K 3.01K 6.05M     9   584 proc
6.96K   646   315  152K 24.4K  82.5K    70 45.4K 83.1M  205K 3.00G tmpfs
    0     0     0 13.3K     0  33.2K   126    14 2.83K     0     0 zfs
    0     0     0 6.26K     0  14.4K   303    40 17.2K     0     0 proc
    0     0     0   377     0     45     0     0     0     0     0 tmpfs
    0     0     0     0     0      1     0     0     0     0     0 zfs
    0     0     0     2     0      2     2     1   440     0     0 proc
    0     0     0     0     0      0     0     0     0     0     0 tmpfs
^C
root@wacken:~# fsstat -l zfs proc tmpfs 1
 read read   read write write write rddir rddir rddir
  ops bytes  time   ops bytes  time   ops bytes  time
 241K 1019M    4n  111K 3.61G 3.00n 30.4K 10.6M  194n zfs
3.06K 6.07M 3.00n     9   584  261n 63.3K 67.9M  232n proc
45.5K 83.1M  295n  206K 3.01G    0n    70 3.58K 45.0n tmpfs
   14 2.83K 48.0n     0     0    0n     0     0    0n zfs
    0     0    0n     0     0    0n     2 2.27K  168n proc
    0     0    0n     0     0    0n     0     0    0n tmpfs
    0     0    0n     0     0    0n     0     0    0n zfs
    1   440 37.0n     0     0    0n     2 2.30K  218n proc
    0     0    0n     0     0    0n     0     0    0n tmpfs
    0     0    0n     0     0    0n   980  193K  100n zfs
    0     0    0n     0     0    0n    41 10.4K  149n proc
    0     0    0n     0     0    0n     0     0    0n tmpfs
^C

This will be pretty useful from time to time.

SSL/TLS Alert Protocol & the Alert Codes

Alert Code

Alert
Message

Description

0

close_notify

Notifies the recipient that the sender will not send any more messages on this connection.

10

unexpected_message

Received an inappropriate message This alert should never be observed in communication between proper implementations. This message is always fatal.

20

bad_record_mac

Received a record with an incorrect MAC. This message is always fatal.

21

decryption_failed

Decryption of a TLSCiphertext record is decrypted in an invalid way: either it was not an even multiple of the block length or its padding values, when checked, were not correct. This message is always fatal.

22

record_overflow

Received a TLSCiphertext record which had a length more than 2^14+2048 bytes, or a record decrypted to a TLSCompressed record with more than 2^14+1024 bytes. This message is always fatal.

30

decompression_failure

Received improper input, such as data that would expand to excessive length, from the decompression function. This message is always fatal.

40

handshake_failure

Indicates that the sender was unable to negotiate an acceptable set of security parameters given the options available. This is a fatal error.

42

bad_certificate

There is a problem with the certificate, for example, a certificate is corrupt, or a certificate contains signatures that cannot be verified.

43

unsupported_certificate

Received an unsupported certificate type.

44

certificate_revoked

Received a certificate that was revoked by its signer.

45

certificate_expired

Received a certificate has expired or is not currently valid.

46

certificate_unknown

An unspecified issue took place while processing the certificate that made it unacceptable.

47

illegal_parameter

Violated security parameters, such as a field in the handshake was out of range or inconsistent with other fields. This is always fatal.

48

unknown_ca

Received a valid certificate chain or partial chain, but the certificate was not accepted because the CA certificate could not be located or could not be matched with a known, trusted CA. This message is always fatal.

49

access_denied

Received a valid certificate, but when access control was applied, the sender did not proceed with negotiation. This message is always fatal.

50

decode_error

A message could not be decoded because some field was out of the specified range or the length of the message was incorrect. This message is always fatal.

51

decrypt_error

Failed handshake cryptographic operation, including being unable to correctly verify a signature, decrypt a key exchange, or validate a finished message.

60

export_restriction

Detected a negotiation that was not in compliance with export restrictions; for example, attempting to transfer a 1024 bit ephemeral RSA key for the RSA_EXPORT handshake method. This message is always fatal.

70

protocol_version

The protocol version the client attempted to negotiate is recognized, but not supported. For example, old protocol versions might be avoided for security reasons. This message is always fatal.

71

insufficient_security

Failed negotiation specifically because the server requires ciphers more secure than those supported by the client. Returned instead of handshake_failure. This message is always fatal.

80

internal_error

An internal error unrelated to the peer or the correctness of the protocol makes it impossible to continue, such as a memory allocation failure. The error is not related to protocol. This message is always fatal.

90

user_cancelled

Cancelled handshake for a reason that is unrelated to a protocol failure. If the user cancels an operation after the handshake is complete, just closing the connection by sending a close_notify is more appropriate. This alert should be followed by a close_notify. This message is generally a warning.

100

no_renegotiation

Sent by the client in response to a hello request or sent by the server in response to a client hello after initial handshaking. Either of these would normally lead to renegotiation; when that is not appropriate, the recipient should respond with this alert; at that point, the original requester can decide whether to proceed with the connection. One case where this would be appropriate would be where a server has spawned a process to satisfy a request; the process might receive security parameters (key length, authentication, and so on) at start-up and it might be difficult to communicate changes to these parameters after that point. This message is always a warning.

255

unsupported_extension