# proxy-verifier
**Repository Path**: mirrors_yahoo/proxy-verifier
## Basic Information
- **Project Name**: proxy-verifier
- **Description**: Proxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP proxies. It builds a verifier-client binary and a verifier-server binary which each read a set of YAML or JSON files that specify the HTTP traffic for the two to exchange.
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-08-19
- **Last Updated**: 2026-11-04
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
Table of Contents
=================
* [Table of Contents](#table-of-contents)
* [Proxy Verifier](#proxy-verifier)
* [Traffic Replay Specification](#traffic-replay-specification)
* [HTTP Specification](#http-specification)
* [Server Response Lookup](#server-response-lookup)
* [HTTP/2 Specification](#http2-specification)
* [HEADERS and DATA frame](#headers-and-data-frame)
* [RST_STREAM frame](#rst_stream-frame)
* [GOAWAY frame](#goaway-frame)
* [Await](#await)
* [Protocol Specification](#protocol-specification)
* [PROXY protocol support](#proxy-protocol-support)
* [Session and Transaction Delay Specification](#session-and-transaction-delay-specification)
* [Content Delay Specification](#content-delay-specification)
* [Keep Connection Open](#keep-connection-open)
* [Traffic Verification Specification](#traffic-verification-specification)
* [Request Presence Verification](#request-presence-verification)
* [Field Verification](#field-verification)
* [Request Method Verification](#request-method-verification)
* [URL Verification](#url-verification)
* [Status Verification](#status-verification)
* [Body Verification](#body-verification)
* [Example Replay File](#example-replay-file)
* [Installing](#installing)
* [Prebuilt Binaries](#prebuilt-binaries)
* [Building from Source](#building-from-source)
* [Prerequisites](#prerequisites)
* [Build](#build)
* [Development Docker Images](#development-docker-images)
* [Install](#install)
* [ASan Instrumentation](#asan-instrumentation)
* [Debug Build](#debug-build)
* [Portable Build](#portable-build)
* [Native Build](#native-build)
* [Running the Tests](#running-the-tests)
* [Unit Tests](#unit-tests)
* [Uranium Tests](#uranium-tests)
* [Usage](#usage)
* [Required Arguments](#required-arguments)
* [Optional Arguments](#optional-arguments)
* [--format <format-specification>](#--format-format-specification)
* [--keys <key1 key2 ... keyn>](#--keys-key1-key2--keyn)
* [--verbose](#--verbose)
* [--interface <interface>](#--interface-interface)
* [--no-proxy](#--no-proxy)
* [--strict](#--strict)
* [--allow-unprocessed-verifications](#--allow-unprocessed-verifications)
* [--rate <requests/second>](#--rate-requestssecond)
* [--repeat <number>](#--repeat-number)
* [--run-continuously](#--run-continuously)
* [--thread-limit <number>](#--thread-limit-number)
* [--qlog-dir <directory>](#--qlog-dir-directory)
* [--tls-secrets-log-file <secrets_log_file_name>](#--tls-secrets-log-file-secrets_log_file_name)
* [--send-buffer-size <size>](#--send-buffer-size-size)
* [--poll-timeout <timeout_ms>](#--poll-timeout-timeout_ms)
* [--io-uring <mode>](#--io-uring-mode)
* [Tools](#tools)
* [Replay Gen replay-gen.py](tools/replay-gen.py)
* [-n,--number <NUMBER>](#-n--number-number)
* [-tl,--trans-lower <TRANS_LOWER>](#-tl--trans-lower-trans_lower)
* [-tu,--trans-upper <TRANS_UPPER>](#-tu--trans-upper-trans_upper)
* [-sl,--sess-lower <SESS_LOWER>](#-sl--sess-lower-sess_lower)
* [-su,--sess-upper <SESS_UPPER>](#-su--sess-upper-sess_upper)
* [-tp,--trans-protocols <TRANS_PROTOCOLS>](#-tp--trans-protocols-trans_protocols)
* [-u,--url-file <URL_FILE>](#-u--url-file-url_file)
* [-o,--output <OUTPUT>](#-o--output-output)
* [-p,--prefix <PREFIX>](#-p--prefix-prefix)
* [-j,--out-json](#-j--out-json)
* [Remap Config to URL List remap-config-to-url-list.py](tools/remap-config-to-url-list.py)
* [-o,--output <OUTPUT_FILE>](#-o--output-output_file)
* [--no-ip](#--no-ip)
* [Contribute](#contribute)
* [License](#license)
# Proxy Verifier
Proxy Verifier is an HTTP replay tool designed to verify the behavior of HTTP
proxies. It builds a verifier-client binary and a verifier-server binary which
each read a set of YAML files (or JSON files, since JSON is YAML) that specify
the HTTP traffic for the two to exchange.
Proxy Verifier supports the HTTP replay of the following protocols:
* HTTP/1.x and HTTP/2
* HTTP and HTTP over TLS
* IPv4 and IPv6
Broadly speaking, Proxy Verifier is designed to address two proxy testing
needs:
* **Traffic correctness testing**: In this context Proxy Verifier can be used
in manual or automated end to end tests to verify correct behavior of the
details of generally a small number of transactions. [Transaction
Box](https://github.com/SolidWallOfCode/txn_box) is an example of a tool that
relies entirely on Proxy Verifier for its automated end to end tests.
* **Production simulation testing**: In this context, Proxy Verifier is used in
an isolated lab environment configured as much like a production environment
as possible. The Verifier client and server are provided replay files as
input that are either auto generated or collected via a tool like [Traffic
Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)
from an actual production environment. Proxy verifier then replays this
production-like traffic against the proxy under test. Proxy Verifier can
replay such traffic at rates over 10k requests per second. Here are some
examples where this use of Proxy Verifier can be helpful:
* Safely testing experimental versions of a patch.
* Running diagnostic versions of the proxy that may not be performant enough
for the production environment. Debug, Valgrind, and ASan builds are
examples of build configurations whose resultant proxy performance may be
impossible to run in production but can be run safely in the production
simulation via Proxy Verifier.
* Performance comparison across versions of the proxy. Since Proxy Verifier
replays the traffic for the same set of replay files consistently across
runs, different versions of the proxy can be compared with regard to their
performance against the same replay dataset. This affords the ability to
compare detect performance improvement or degradation across versions of the
proxy.
It should be noted that when using Proxy Verifier in a production simulation
environment, the proxy will be receiving traffic from the client that looks
like production traffic. Thus, by design, the HTTP request targets and Host
header fields will reference actual production servers. It will be important to
configure the proxy under test to direct these requests not to the actual
production servers but to the Proxy Verifier server or servers. The way this
is achieved will depend upon the proxy. One way to accomplish this is to
configure the production environment with a test DNS server such as
MicroDNS configured to resolve all host names to the Proxy Verifier server or
servers in the production simulation environment.
## Traffic Replay Specification
### HTTP Specification
Proxy Verifier traffic behavior is specified via YAML files. The behavior for
each connection is specified under the top-most `sessions` node, the value of
which is a sequence where each item in the sequence describes the
characteristics of each connection. For each session the protocol stack can be
specified via the `protocol` node. This node describes the protocol
characteristics of the connection such as what version of HTTP to use, whether
to use TLS and what the characteristics of the TLS handshake should be. In the
absence of a `protocol` node the default protocol is HTTP/1 over TCP. See
[Protocol Specification](#protocol-specification) below for details about the
`protocol` node. Each session is run in parallel by the verifier-client
(although see [--thread-limit <number>](#--thread-limit-number) below for
a way to serialize them).
In addition to protocol specification, each session in the `sessions`
sequence contains a `transactions` node. This itself is a sequence of
transactions that should be replayed for the associated session. Within each
transaction, Proxy Verifier's traffic replay behavior is specified in the
`client-request` and `server-response` nodes. `client-request` nodes are used
by the Proxy Verifier client and tell it what kind of HTTP requests it should
generate. `server-response` nodes are used by the Proxy Verifier server to
indicate what HTTP responses it should generate. Proxy traffic verification
behavior is described in the `proxy-request` and `proxy-response` nodes which
will be covered in [Traffic Verification
Specification](#traffic-verification-specification). For HTTP/1, these
transactions are run in sequence for each session; for HTTP/2, the transactions
are run in parallel.
For HTTP/1 requests, `client-request` has a map as a value which contains each of
the following key/value YAML nodes:
1. `method`: This takes the HTTP method as the value, such as `GET` or `POST`.
1. `url`: This takes the request target as the value, such as `/a/path.asp` or
`https://www.example.com/a/path.asp`.
1. `version`: This takes the HTTP version, such as `1.1` or `2`.
1. `headers`: This takes a `fields` node which has, as a value, a sequence of
HTTP fields. Each field is itself a sequence of two values: the name of the
field and the value for the field.
1. `content`: This specifies the body to send. It takes a map as a value. The
user can specify a `size` integer value in which an automated body of that
size will be generated. Otherwise a `data` string value can be provided in
which the specified body will be sent and an `encoding` taking `plain` or
`uri` to specify that the string is raw or URI encoded.
Here's an example of a `client-request` node describing an HTTP/1.1 POST
request with a request target of `/pictures/flower.jpeg`
and containing four header fields with a body size of 399 bytes:
```YAML
client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, 1234 ]
content:
size: 399
```
For convenience, if Proxy Verifier detects a Content-Length header, then the
`content` node stating the size of the body is not required. Thus this can be
simplified to:
```YAML
client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, 1234 ]
```
`server-response` nodes indicate how the Proxy Verifier server should respond
to HTTP requests. In place of `method`, `url`, and `version`, HTTP/1 responses
take the following nodes:
1. `status`: This takes an integer corresponding to the HTTP response status,
such as `200` or `404`.
1. `reason`: This takes a string that describes the status, such as `"OK"` or
`"Not Found"`.
1. `on_connect`: This controls the transport behavior after the request is
read. Supported values are `accept` (default), `refuse`, and `reset`.
Here's an example of an HTTP/1 `server-response` with a status of 200, four
fields, and a body of size 3,432 bytes:
```YAML
server-response:
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
size: 3432
```
Observe in this case that the response contains the `Transfer-Encoding:
chunked` header field, indicating that the body should be chunk encoded. Proxy
Verifier supports chunk encoding for both requests and responses and will use
this when it detects the `Transfer-Encoding: chunked` header field. In this
case, the 3,432 bytes will be the size of the chunked body payload without the
bytes for the chunk protocol (chunk headers, etc.).
Finally, here is an example of a response with specific body content sent (YAML
in this case) as opposed to the generated content specified by the
`content:size` nodes above:
```YAML
server-response:
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, text/yaml ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
encoding: plain
data: |
### Heading
* Bullet
* Points
```
The `on_connect` directive can be used to simulate origin-side connection
failures without sending an HTTP response. This is useful when testing proxy
behavior for upstream errors. Supported values are:
* `accept`: Normal behavior. The Verifier server sends the configured HTTP
response. This is the default. At the transport level, the connection stays
open and no special close behavior is injected.
* `refuse`: The Verifier server closes the upstream connection after it reads
and validates the request, without sending a response. For TCP-based
protocols such as HTTP/1 and HTTP/2 over TLS, this is a normal close, so the
peer observes the connection being shut down with a FIN rather than an HTTP
response.
* `reset`: The Verifier server aborts the upstream connection after it reads
and validates the request, without sending a response. For TCP-based
protocols such as HTTP/1 and HTTP/2 over TLS, this is an abortive close: the
server enables linger-for-reset before closing so the peer observes a TCP
RST instead of a FIN or an HTTP response.
These `refuse` and `reset` descriptions are specifically about TCP transport
behavior. QUIC does not use TCP, so it does not have TCP FIN or RST packets.
The analogous QUIC behaviors would involve stream resets or connection-close
frames rather than TCP teardown. Proxy Verifier does not currently implement
server-side HTTP/3/QUIC, so `on_connect` does not presently map to QUIC-
specific close semantics.
When `on_connect` is set to `refuse` or `reset`, the `status` field becomes
optional because no response is sent. For example:
```YAML
server-response:
on_connect: refuse
```
#### Server Response Lookup
The `client-request` and `server-response` nodes are all that is required to
direct Proxy Verifier's replay of a transaction. The next section will describe
how to specify proxy transaction verification behavior via the `proxy-request` and
`proxy-response` nodes. Before proceeding to that, however, it is valuable to
understand how the Verifier server decides which response to send for any given
request it receives. Consider again the `client-request` node in the preceding
example:
```YAML
client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, 1234 ]
content:
size: 399
```
Note the presence of the `uuid` field. A field such as this is sent by the
client and used by the server. When a server receives a request, keep in mind
that all it has to direct its behavior is what it has parsed from the replay
file or files and what it sees in the request. There may be hundreds of
thousands of parsed `server-response` nodes, each describing a unique response
with which the server may reply for any given incoming request. The server does
not talk directly to the client, so it cannot communicate with it and say, "Hey
Verifier client, I just read such and such request off the wire. Which response
would you like me to send for this?" When it receives an HTTP request,
therefore, it only has the contents of that request from which to choose its
response. To facilitate the server's behavior, a unique *key* is associated
with every request. During the YAML parsing phase, the Verifier server
associates each parsed `server-response` with a key it derives from the
associated `client-request` for the transaction. During the traffic replay
phase, when it reads a request off the wire, it derives a key from the HTTP
request using the same process for generating keys from `client-request` nodes
used in the parsing phase. It then looks up the proper YAML-specified response
using that key. By default, Proxy Verifier uses the `uuid` HTTP header field
values as the key, as utilized in the examples in this document, but this is
configurable via the `--format` command line argument. For details see the
section describing this argument below.
Both the client and the server will fail the YAML parsing phase and exit with a
non-zero return if either parses a transaction for which they cannot derive a
key. If during the traffic processing phase the Verifier server somehow
receives a request for which it cannot derive a key, it will return a *404 Not
Found* response and close the connection upon which it received the request.
### HTTP/2 Specification
For HTTP/2, the protocol describes the initial request line values with pseudo
header fields. For that protocol, therefore, the user need not specify the
`method`, `url`, and `version` nodes and would instead specify these values
more naturally as pseudo headers in the `fields` sequence. Here is an example of
an HTTP/2 `client-request` analogous to the HTTP/1 request above (note that the
`Content-Length` header field is optional in HTTP/2 and as such is omitted here):
```YAML
client-request:
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, 1234 ]
content:
size: 399
```
The status code is described in the `:status` pseudo header field.
Also, HTTP/2 does not allow for chunk encoding nor does it use the `Connection`
header field, using instead its framing mechanism to describe the body and
session lifetime. An analogous HTTP/2 response to the HTTP/1 request above,
therefore, would look like the following:
```YAML
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
content:
size: 3432
```
Trailer response headers are useful for transmitting additional fields after the
message body, providing dynamically generated metadata such as integrity checks
or post-processing status. Proxy Verifier supports sending and receiving trailer
headers, as demonstrated below:
```YAML
server-response:
status: 200
headers:
fields:
# Some fields...
content:
# Some data...
trailers:
encoding: esc_json
fields:
- [ x-test-trailer-1, one ]
- [ x-test-trailer-2, two ]
```
It is also possible to specify everything as a sequence of frames. The available
options for the frame sequence are:
* `DATA`
* `HEADERS`
* `RST_STREAM`
* `GOAWAY`
Here's an example replay file:
```YAML
client-request:
frames:
- HEADERS:
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, 1234 ]
- DATA:
content:
size: 399
- RST_STREAM:
error-code: STREAM_CLOSED
server-response:
frames:
- HEADERS:
headers:
fields:
- [ :status, 200 ]
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- DATA:
content:
size: 3432
```
#### HEADERS and DATA frame
The `HEADERS` and `DATA` frame nodes are designed to be specified in a way that is consistent
with their HTTP/1 counterparts. From a parsing perspective, this means they simply wrap the
`headers` and `content` nodes that are used for HTTP/1 specification as described above. For
an example, see the `RST_STREAM` frame section below.
Note that multiple `DATA` frames can be specified for requests and responses. The replay follows
the same order the `DATA` frames are listed. See the example below:
```YAML
client-request:
frames:
- HEADERS:
headers:
fields:
- [:method, POST]
- [:scheme, https]
- [:authority, example.data.com]
- [:path, /a/path]
- [Content-Type, text/html]
- [uuid, 1]
- DATA:
content:
encoding: plain
data: client_data_1
- DATA:
content:
encoding: plain
data: client_data_2
```
#### RST_STREAM frame
In some cases, there might be a need to test the behavior of the proxy when the
client or server terminates the stream, either because of an unexpected error or intentionally
cancelling the stream. In HTTP/2, peers terminate a transaction by sending a `RST_STREAM` frame
which includes an error code. In the replay file, this is specified via a `RST_STREAM` frame
node which is ordered within the stream to indicate when it should be sent and includes an
`error-code` node to specify which error code should be used.
The following sample replay file snippet demonstrate how to specify such a test scenario:
```YAML
sessions:
- protocol:
stack: http2
tls:
sni: test_sni
version: 4
transactions:
- client-request:
frames:
- HEADERS:
headers:
fields:
- [:method, POST]
- [:scheme, https]
- [:authority, example.data.com]
- [:path, /a/path]
- [Content-Type, text/html]
- [Content-Length, '11']
- [uuid, 1]
- RST_STREAM:
error-code: INTERNAL_ERROR
- DATA:
content:
encoding: plain
data: client_test
size: 11
```
Note that this example specifies the following:
* A sequence of frames to be sent under the `client-request` node. The order
of the frames listed is the order of the frames that Proxy Verifier will send during
the traffic replay of the stream.
* Thus, in this case, the client terminates the stream after sending the `HEADERS` frame since
the `RST_STREAM` is specified after the `HEADERS` frame in the sequence.
* The client terminates with the error code `INTERNAL_ERROR`, specified by `error-code` under
the `RST_STREAM` frame.
Be aware that when Proxy Verifier generates HTTP/2 frames, it determines from the frames elements where to
insert the `END_STREAM` flag. If a request or response just has a `HEADERS` frame, then the `END_STREAM`
flag will be added to the end of the `HEADERS` frame. If there are both `HEADERS` and DATA frames, then the
`END_STREAM` will be placed after the `DATA` frame. Proxy Verifier does not, however, use `RST_STREAM`
frames to influence the `END_STREAM` flag. Thus in the above example, having the `RST_STREAM` between the
`HEADERS` and `DATA` frames means that the `HEADERS` will not have an `END_STREAM` flag before the
`RST_STREAM` is sent. For this reason, you must include a `DATA` frame after a `RST_STREAM` frame, even
though the `DATA` frame will not be sent, in order to keep the stream open at the time the `RST_STREAM` is
sent.
The server side stream termination can be set in the same way, but under the `server-response` node.
The available options for `error-code` are:
* NO_ERROR
* PROTOCOL_ERROR
* INTERNAL_ERROR
* FLOW_CONTROL_ERROR
* SETTINGS_TIMEOUT
* STREAM_CLOSED
* FRAME_SIZE_ERROR
* REFUSED_STREAM
* CANCEL
* COMPRESSION_ERROR
* CONNECT_ERROR
* ENHANCE_YOUR_CALM
* INADEQUATE_SECURITY
* HTTP_1_1_REQUIRED
Finer grained frame ordering can also be specified via a `delay` node which specifies the time
delay before sending the `RST_STREAM` frame. For example, to delay sending a `RST_STREAM` frame
for 1 second, specify a `delay: 1s` directive like so:
```YAML
- RST_STREAM:
error-code: INTERNAL_ERROR
delay: 1s
```
A detailed description of the `delay` node can be found [here](#session-and-transaction-delay-specification).
#### GOAWAY frame
The `GOAWAY` frame acts similar to the `RST_STREAM` frame shown above. However, rather than terminating
a stream, `GOAWAY` frame terminates the connection. It supports `error-code` and `delay` as described
for `RST_STREAM` frame above.
HTTP/2 sessions also have a `close-on-goaway` directive. This boolean
configuration informs how the Verifier client should behave when there are more
streams configured in the replay YAML file after a `GOAWAY` frame is received
(see [RFC 7540 section
6.8](https://datatracker.ietf.org/doc/html/rfc7540#section-6.8) for details
about the `GOAWAY` frame). When set to `true`, on receipt of a `GOAWAY` frame,
the client terminates the connection after processing the streams that are
currently being processed. When set to `false`, the client does not terminate
the connection and continues with subsequent streams specified in the replay
file. The default value of this option is `true` since that is in keeping with
the RFC. Here's an example session configured with `close-on-goaway` set to
`false`:
```YAML
sessions:
- protocol:
stack: http2
tls:
sni: test_sni
version: 4
close-on-goaway: false
transactions:
```
#### Await
By protocol specification, HTTP/1 transactions are serialized. That is, apart
from the rarely used pipelined request protocol feature (which is not supported
by Proxy Verifier), each HTTP/1 transaction in a connection is not exchanged
until the request and response of the previous one is completed. By contrast,
the HTTP/2 protocol specifies that streams (i.e., HTTP/2 transactions) are
multiplexed within their sessions (i.e., HTTP/2 connections). For the Verifier
client, this means that, by default, requests for streams within a session are
each sent immediately in the order that they are specified in the replay file
without waiting for their respective responses. That is, if a replay file
specifies three streams within a session, then the client will send the request
for each stream as quickly as it can serialize it on the socket for the
session before waiting for any responses.
The timing for sending the stream requests can be modified from this default
behavior in several ways. As with replaying HTTP/1 traffic, if the streams
contain timing specification nodes, then the client can be configured to replay
the streams at a rate scaled to that timing information. See the documentation
for the [--rate](#--rate-requestssecond) argument for a description of how
this works. Also, the user can include
[delay](#session-and-transaction-delay-specification) nodes to space out the
replay of each stream.
In addition to these configurations supported in HTTP/1 traffic replay, for
HTTP/2 and HTTP/3 the Verifier client supports the `await` node to control when
it starts streams. This node takes a transaction key or a sequence of
transaction keys, the responses of which are dependencies for sending the request
for the associated stream. That is, a stream with an `await` node will not be
sent until all the responses are received for the listed keys.
As an example, consider the specification of the following session with two
streams:
```YAML
# Specify an HTTP/2 session
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
transactions:
# Stream 1
- client-request:
headers:
fields:
- [ :method, GET ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, first-stream ]
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Content-Type, image/jpeg ]
- [ X-Response, first-response ]
content:
size: 3432
# Stream 2
- client-request:
# This await will cause the Verifier client to hold off on sending this
# request until the response to first-stream is received.
await: first-stream
headers:
fields:
- [ :method, GET ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, second-stream ]
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Content-Type, image/jpeg ]
- [ X-Response, second-response ]
content:
size: 3432
```
Notice that the second transaction with key `second-stream` contains an
`await` node. Without this node, the Verifier client would send both requests
back to back. That is, the `second-stream` request would be sent immediately
after the `first-stream` request was sent. With the `await` node specifying a
dependency upon the `first-stream` transaction, however, the Verifier client
will instead delay sending the `second-stream` request until the response to
`first-stream` was received from the proxy.
There are a few things related to `await` nodes to be cognizant of:
* `await` nodes can be used in conjunction with `delay` nodes for
`client-request` specifications. When both are used, the Verifier client will
first hold off on sending the request until all dependant responses specified
by the keys in `await` are processed. Once the dependant responses are
received, then the `delay` node is processed and the request is further
delayed for the amount of time specified by the value of the `delay` node.
Then the request is sent after the delay.
* As stated before, the Verifier client is designed to processes streams and
their directives in the order that they are specified in the `transactions`
sequence for their stream. This means that any `await` or `delay` nodes for
one transaction causes a corresponding delay for later nodes before they are
processed. That is to say, if a session is specified with five streams, and
the third stream has an `await` specified for the first two streams, then the
fourth and fifth streams will also be delayed behind the third stream while
it awaits the responses for the first and second streams, even if streams
four and five contain no `await` nor `delay` nodes themselves.
* Transaction keys specified via `await` can only be for earlier transactions
in the same session that the `client-request` node is in. That is, the
Verifier client does not support a stream in one session to `await` the
response to a stream in a different session.
### Protocol Specification
The above discussed the replay YAML nodes that describe how Proxy Verifier will
craft HTTP layer traffic. This section discusses how the user specifies the
lower layer protocols used to transport this HTTP traffic.
As stated above, each HTTP session is described as an item under the `sessions`
node sequence. Each session takes a map. HTTP transactions are described under
the `transactions` key described above. In addition to `transactions`, a
session also takes an optional `protocol` node. This node can be specified in
either of two forms:
1. A concise map form, recommended for human-written replay files.
1. A verbose sequence form, which is still useful for tools that dump the
protocol stack explicitly.
The concise map form uses a `stack` key for common defaults. The rest of the
map can then selectively include the already defined protocol layer maps as
overrides. For example:
```YAML
sessions:
- protocol:
stack: http2
tls:
sni: test_sni
transactions:
# ...
```
This selects the HTTP/2 stack and then overrides the TLS layer to apply the SNI
of `test_sni`. If you omit override maps such as `tls`, then the selected stack
is used with its default behavior.
The supported concise map keys are:
| Key | Supported Values | Description
| ----- | ---------------- | -----------
| stack | `http`, `https`, `http2`, `http3` | Shorthand for common protocol stacks.
| tls | map | Optional TLS layer override. Supports the same TLS keys as the verbose form, such as `sni`, `request-certificate`, `proxy-provided-certificate`, `verify-mode`, and `alpn-protocols`.
| proxy-protocol | `1`, `2`, or map | A PROXY protocol version shorthand or a map with `version`, `src-addr`, and `dst-addr`.
| http | scalar version or map | Optional explicit HTTP layer override when not using `stack`.
| tcp | map | Optional explicit TCP layer override.
| ip | map | Optional explicit IP layer override.
The verbose sequence form takes an ordered sequence of maps, where each item in
the sequence describes the characteristics of a protocol layer. The sequence is
expected to be ordered from higher layer protocols (such as HTTP and TLS) to
lower layer protocols (such as IP).
Here is the equivalent verbose protocol node along with `sessions` and
`transactions` nodes provided to give some context:
```YAML
sessions:
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
transactions:
# ...
```
Note again how the `protocol` node is under the `sessions` node which takes a
sequence of sessions. This sample shows the start of a single session that, in
this case, provides a protocol description via a `protocol` key. This same
session also has a truncated set of transactions that will be specified under
the `transactions` key. Looking further at the `protocol` node, observe that
this session has four layers described for it: http, tls, tcp, and ip. The
`http` node specifies that the session should use the HTTP/2 protocol. The
`tls` node specifies that the client should use an SNI of "test\_sni" in the
TLS client hello handshake. Further, this should be transported over TCP on IP.
The following verbose `name` values are supported for `protocol`:
| Name | Node | Supported Values | Description
| ----- |-------- | ---------------- | -----------
| http | | |
| | version | {1, 2} | Whether to use HTTP/1 or HTTP/2.
| tls | | |
| | sni | string | The SNI to send in the TLS handshake.
| | request-certificate | boolean | Whether the client or server should request a certificate from the proxy.
| | proxy-provided-certificate | boolean | This directs the same behavior as the request-certificate directive. This alias is helpful when the node describes what happened in the past, such as in the context of a replay file specified by [Traffic Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html).
| | verify-mode | {0-15} | The value to pass directly to OpenSSL's `SSL_set_verify` to control peer verification in the TLS handshake. This allows fine grained control over TLS verification behavior. `0` corresponds with SSL_VERIFY_NONE, `1` corresponds with SSL_VERIFY_PEER, `2` corresponds with SSL_VERIFY_FAIL_IF_NO_PEER_CERT, `4` corresponds with SSL_VERIFY_CLIENT_ONCE, and `8` corresponds with SSL_VERIFY_POST_HANDSHAKE. Any bitwise OR'd value of these values can be provided. For details about their behavior, see OpenSSL's [SSL_verify_cb](https://www.openssl.org/docs/man1.1.1/man3/SSL_verify_cb.html) documentation.
| | alpn-protocols | sequence of strings | This specifies the server's protocol list used in ALPN selection. See OpenSSL's [SSL_select_next_proto](https://www.openssl.org/docs/man1.0.2/man3/SSL_select_next_proto.html) documentation for details.
| proxy-protocol | | |
| | version | {1, 2} | Whether to use PROXY header v1 or v2
| | src-addr | string | The source address and port in the PROXY header. Specified in the format of `111.111.111.111:11111`.
| | dst-addr | string | The destination address and port in the PROXY header. Specified in the same format of `src-addr`.
| tcp | | |
| ip | | |
The following protocol specification features are not currently implemented:
* HTTP/2 is only supported over TLS. Proxy Verifier uses ALPN in the TLS
handshake to negotiate HTTP/2 with the proxy. HTTP/2 upgrade from HTTP/1
without TLS is not supported.
* The user cannot supply a TLS version to negotiate for the handshake.
Currently, if the TLS node is present, Proxy Verifier will use the highest
TLS version it can negotiate with the peer. This is OpenSSL's default
behavior. An enhancement request to support A TLS version specification
feature request is recorded in issue
[101](https://github.com/yahoo/proxy-verifier/issues/101).
* Similarly, the user cannot specify whether to use IPv4 or IPv6 via YAML.
Proxy Verifier can test IPv6, but it does so via the user passing IPv6
addresses on the command line. An IP version feature request is recorded in
issue [100](https://github.com/yahoo/proxy-verifier/issues/100).
* Only TCP is supported. There have been recent discussions about adding
HTTP/3 support, which is over UDP, but work for that has not yet started.
* The PROXY protocol support is limited to the `PROXY` command via TCP over IPv4 or IPv6. Therefore,
* No support for `AF_UNIX` socket address as either source and destination address.
* No support for UDP transport type.
* No support for `LOCAL` command(in v2 header).
If there is no `protocol` node specified, then Proxy Verifier will default to
establishing an HTTP/1 connection over TCP (no TLS).
#### PROXY protocol support
Generally speaking, a server sitting downstream from a proxy does not have
visibility into the client's network socket information that lies behind the
proxy. PROXY Protocol is a mechanism to provide visibility for this. PROXY
protocol is a network protocol that communicates a client's source and
destination IP and port information via a set of bytes at the start of a TCP
connection from a proxy. Here is a link to the protocol description:
https://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt
Proxy Verifier supports sending and receiving the PROXY protocol, which can be
helpful to verify the PROXY protocol behavior of the proxy under test.
The feature can be enabled by specifying the `proxy-protocol` protocol node as
outlined above. If enabled, the Verifier client would send out the PROXY
protocol header at the beginning of the connection. Upon receiving a PROXY
protocol header, the Verifier server would display it in the human-readable v1
format. Note that the specification of source and destination addresses are
supported but not required; if not specified, the Proxy Verifier client would
send out PROXY message with the source and destination addresses matching the
underlying socket, similar to how `curl --haproxy-protocol` behaves.
### Session and Transaction Delay Specification
A user can also stipulate per session and/or per transaction delays to be
inserted by the Verifier client and server during the replay of traffic. This
is done via the `delay` node which takes a unit-specified duration for the
associated delay. During traffic replay, the delay is inserted before the
associated session is established or before the client request or server
response is sent. Proxy Verifier recognizes the following time duration units
for the `delay` node:
Unit Suffix | Meaning
----------- | -------
s | seconds
ms | milliseconds
us | microseconds
Here is a sample replay file snippet that demonstrates the specification of a
set of delays:
```YAML
sessions:
- delay: 2s
transactions:
client-request:
delay: 15ms
method: POST
url: /a/path.jpeg
version: '1.1'
headers:
fields:
- [ Content-Length, '399' ]
- [ Content-Type, image/jpeg ]
- [ Host, example.com ]
- [ uuid, 1 ]
server-response:
delay: 17000 us
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
size: 3432
```
Note that this example specifies the following delays:
* The client delays 2 seconds before establishing the session.
* The client also delays 15 milliseconds before sending the client
request.
* The server delays 17 milliseconds (17,000 microseconds) before sending the
corresponding response after receiving the request. This same delay is
applied before `server-response.on_connect: refuse` or `reset` actions.
Be aware of the following characteristics of the `delay` node:
* The Verifier client interprets and implements `delay` for sessions in the
`sessions` node and for transactions in the `client-request` node. The
Verifier server interprets delay only for transactions in the
`server-response` node and ignores `sessions` delays. Since the server is
passive in receiving connections, it's not obvious what a server-side session
delay would mean in this context.
* Notice that Proxy Verifier supports microsecond level delay granularity, and
does indeed faithfully insert delays at the appropriate times during replay
with that precision of time. Be aware, however, that for the vast majority of
networks anything more precise than a millisecond will not generally be useful.
See also [--rate <requests/second>](#--rate-requestssecond) below for
rate specification of transactions.
### Content Delay Specification
The `delay` node described above is applied before a message is sent. To pause
in the middle of a message instead, a `delay` node can be placed inside a
`content` node. The message headers are written, the delay is inserted, and
only then is the body written. This is useful for exercising how a proxy
handles an origin which starts a response and then stalls, such as verifying
that the proxy's timeouts fire at the point they should.
```YAML
server-response:
status: 200
reason: OK
headers:
fields:
- [ Content-Length, '3432' ]
content:
size: 3432
delay: 5s
```
The value uses the same unit-suffixed duration format described in [Session and
Transaction Delay Specification](#session-and-transaction-delay-specification).
This works for HTTP/1.x, HTTP/2, and HTTP/3. For HTTP/1.x and HTTP/2 it applies
to both request and response bodies. Proxy Verifier does not implement
server-side HTTP/3, so over HTTP/3 only a request body can carry a content
delay. For HTTP/2 and HTTP/3 the headers are put on the wire in their `HEADERS`
frame, the body is withheld from the protocol library for the duration of the
delay, and the stream is then resumed so the body follows in its own `DATA`
frame.
Be aware of the following characteristics of a `content` `delay` node:
* An HTTP/2 message with an explicit `frames` node expresses the same behavior
with a `delay` on its `DATA` frame (see [HEADERS and DATA
frame](#headers-and-data-frame)). Specifying a `content` `delay` in a message
which also has a `frames` node is therefore rejected as a replay file error.
* A `content` `delay` composes with a transaction `delay`. A transaction which
specifies both waits before its headers and again before its body.
* Incoming traffic continues to be processed during an HTTP/2 or HTTP/3 content
delay, so flow control and other peer-initiated frames are not stalled behind
it. The HTTP/1.x delay is a plain wait, since there is nothing to multiplex.
* For HTTP/1.x, the delay is inserted whenever a body write follows the headers,
including when that body is empty. For HTTP/2 and HTTP/3 there is no `DATA`
frame to hold back when the body is empty, so the delay is not inserted.
* The delay is not inserted for a request carrying `Expect: 100-continue`, since
no body is written at that point.
* If the peer closes the connection during the delay, which is the expected
outcome when the delay is being used to trigger a proxy timeout, the
subsequent body write fails and is reported. This does not by itself cause
the Verifier server to exit with a non-zero status.
### Keep Connection Open
In certain special situations, a user might need to keep the connection open
after the final transaction in a session is done. To specify how long the connection
needs to be kept open, the user can specify the duration as follows (value format is
the same as [Session and Transaction Delay Specification](#session-and-transaction-delay-specification)):
```YAML
sessions:
- protocol:
- name: http
version: 2
- name: tls
sni: test_sni
- name: tcp
- name: ip
version: 4
keep-connection-open: 2s
transactions:
client-request:
delay: 15ms
method: POST
url: /a/path.jpeg
version: '1.1'
headers:
fields:
- [ Content-Length, '399' ]
- [ Content-Type, image/jpeg ]
- [ Host, example.com ]
- [ uuid, 1 ]
server-response:
delay: 17000 us
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
content:
size: 3432
```
## Traffic Verification Specification
In addition to replaying HTTP traffic as described above, Proxy Verifier also
implements proxy traffic verification. For any given transaction, Proxy
Verifier can optionally verify the HTTP/1 request method, HTTP request target
parts, the response status, and the content of HTTP fields for requests and
responses. Proxy Verifier also supports field verification of HTTP/2 and
HTTP/3 pseudo header fields.
Whereas `client-request` and `server-response` nodes direct the Verifier client
and server (respectively) on how to send traffic, `proxy-request` and
`proxy-response` nodes direct the server and client (respectively) on how to
verify the traffic it receives from the proxy. Thus:
* `client-request` nodes are used by the Verifier client to direct how it will
generate requests it will send to the proxy.
* `server-response` nodes are used by the Verifier server to direct how it will
generate responses to send to the proxy.
* `proxy-request` nodes are used by the Verifier server to direct how it will
verify requests received from the proxy.
* `proxy-response` nodes are used by the Verifier client to direct how it will
verify responses received from the proxy.
In the event that the proxy does not produce the stipulated traffic according
to a verification directive, the Proxy Verifier client or server that detects
the violation will emit a log indicating the violation and will, upon process
exit, return a non-zero status to the shell.
Traffic verification is an optional feature. Thus if Proxy Verifier is being
used simply to replay traffic and the verification features are not helpful,
then the user can simply omit the `proxy-request` and `proxy-response` nodes
and no verification will be performed.
The following sections describe how to specify traffic verification in the YAML
replay file.
### Request Presence Verification
In some situations it is desired to verify whether a particular transaction
does or does not reach the Verifier server. A common example is a cache hit
where the proxy under test should reply directly rather than forwarding the
request upstream. In other cases, it is important to verify that the keyed
request definitely *does* reach the Verifier server.
For these situations, Proxy Verifier supports a transaction-level expectation
under `proxy-request` called `expect`:
```YAML
# Verify the proxy under test replies directly, without proxying this request
# to the verifier-server, for example because ATS serves it from cache.
- client-request:
method: GET
url: /cached/object
version: "1.1"
headers:
fields:
- [ Host, example.com ]
- [ uuid, cache-hit-1 ]
proxy-request:
expect: absent
proxy-response:
status: 200
reason: OK
```
Note the `proxy-request.expect: absent` node. With this directive, the Verifier
server will verify that it never receives a request with the `cache-hit-1`
`uuid`. If the proxy forwards the `cache-hit-1` request to the Verifier server,
the server immediately reports a verification failure and, later upon exit,
will exit with a non-zero shell status.
The supported values are:
Value | Description
-------- | -----------
present | The keyed request must reach the Verifier server. If it never arrives, the server fails at shutdown.
absent | The keyed request must not reach the Verifier server.
If `proxy-request.expect` is omitted, then Proxy Verifier performs no explicit
request-presence verification. In that case, `proxy-request` may still contain
normal request verification directives, and those rules behave as they did
before.
When `proxy-request.expect: absent` is specified, no other verification
directives should be provided. By definition, no request should be received so
there shouldn't be anything to verify. If other verification directives are
provided in addition to `absent` (such as expecting a certain header field or
method), then this is considered a user error and parsing will fail.
When `proxy-request.expect: present` is specified, request arrival is verified
in addition to any normal `proxy-request` verification directives. This allows
the user to require both that the keyed request reaches the Verifier server and
that it matches any configured request verification rules.
For `expect: absent`, `server-response` is optional. If the forbidden request
does arrive and no `server-response` was configured, the Verifier server sends
its built-in `404 Not Found` fallback response in addition to failing
verification. If a specific response is desired in addition to the verification
error, a `server-response` can be provided.
### Field Verification
Recall that in `client-request` and `server-response` nodes, fields are
specified via a sequence of two items: the field name followed by the field
value. For instance, the following incomplete `client-request` node contains
the specification of a single `Content-Length` field (it is incomplete because
it does not specify the method, request target, etc., but this snippet
sufficiently demonstrates a typical description of an HTTP field):
```YAML
client-request:
headers:
fields:
- [ Content-Length, 399 ]
```
For this client request, Proxy Verifier is directed to create a `Content-Length`
HTTP field with a value of `399`.
Field verification is specified in a similar manner, but the second item in the
sequence is a map describing how the field should be verified. The map takes a
`value` item describing the characteristics of the value to verify, if
applicable, and an `as` item providing a directive describing how the field
should be verified. Here is an example of a `proxy-request` node directing the
Verifier server to verify the characteristics of the `Content-Length` field
received from the proxy:
```YAML
proxy-request:
headers:
fields:
- [ Content-Length, { value: 399, as: equal } ]
```
Observe the following from this verification example:
* As described above, notice that the second entry in the field specification
is a map instead of a scalar field value. The Proxy Verifier parser
recognizes this map type as providing a verification specification.
* As with `client-request` and `server-response` field specifications, the
first item in the list describes the field name, in this case
"Content-Length". This indicates that this verification rule applies to the
"Content-Length" HTTP field from the proxy.
* The directive is specified via the `as` key. In this example, `equal` is the
directive for this particular field verification, indicating that the
Verifier server should verify that the proxy's request for this transaction
contains a `Content-Length` field with the exact value of `399`.
Proxy Verifier supports six field verification directives:
Directive | Description
--------- | ------------
absent | The *absence* of a field with the specified field name.
present | The *presence* of a field with the specified field name and having any or no value.
equal | The presence of a field with the specified field name and a value *equal* to the specified value.
contains | The presence of a field with the specified name with a value *containing* the specified value.
prefix | The presence of a field with the specified name with a value *prefixed* with the specified value.
suffix | The presence of a field with the specified name with a value *suffixed* with the specified value.
For all of these field verification behaviors, field names are matched case
insensitively while field values are matched case sensitively.
Thus the following field specification requests no field verification because it does not include a directive and the second item in the sequence is a scarlar:
```YAML
- [ X-Forwarded-For, 10.10.10.2 ]
```
Such non-operative fields can also be specified using the map syntax without an
`as` item:
```YAML
- [ X-Forwarded-For, { value: 10.10.10.2 } ]
```
Fields like this in `proxy-request` and `proxy-response` nodes are permissible
by Proxy Verifier's parser even though they have no functional impact (i.e.,
they do not direct Proxy Verifier's traffic behavior because they are not in
`client-request` nor `server-response` nodes, and they do not describe any
verification behavior). Allowing such fields affords the user the ability to
record the proxy's traffic behavior in situations where field verification is
not required or desired. For example, the [Traffic
Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)
Traffic Server plugin records HTTP traffic and uses these proxy HTTP fields to
indicate what fields were sent by the Traffic Server proxy to the client and
the server. This can be helpful for analyzing the proxy's behavior via these
replay files. Thus this proxy traffic recording function can be helpful to the
user even though Proxy Verifier treats the fields as non-operable.
The following demonstrates the `absent` directive which specifies that the HTTP field `X-Forwarded-For` _with any value_ should not have been sent by the proxy:
```YAML
- [ X-Forwarded-For, { as: absent } ]
```
The following demonstrates the `present` directive which specifies that the HTTP field `X-Forwarded-For` _with any value_ should have been sent by the proxy:
```YAML
- [ X-Forwarded-For, { as: present } ]
```
Notice that for both the `absent` and the `present` directives, the `value` map item is not relevant and need not be provided and, in fact, will be ignored by Proxy Verifier if it is provided.
The following demonstrates the `equal` directive which specifies that `X-Forwarded-For` should have been received from the proxy with the exact value "10.10.10.2":
```YAML
- [ X-Forwarded-For, { value: 10.10.10.2, as: equal } ]
```
The following demonstrates the `contains` directive which specifies that `X-Forwarded-For` should have been received from the proxy containing the value "10" at any position in the field value:
```YAML
- [ X-Forwarded-For, { value: 10, as: contains } ]
```
The following demonstrates the `prefix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value starting with "1":
```YAML
- [ X-Forwarded-For, { value: 1, as: prefix } ]
```
The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value ending with "2":
```YAML
- [ X-Forwarded-For, { value: 2, as: suffix } ]
```
Proxy Verifier supports inverting the result of any rule by using `not` instead of `as`. The following demonstrates the `prefix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value not starting with "a":
```YAML
- [ X-Forwarded-For, { value: a, not: prefix } ]
```
Proxy Verifier also supports ignoring the upper/lower case distinction with another directive: `case: ignore`. The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value starting with "a" or "A":
```YAML
- [ X-Forwarded-For, { value: a, as: prefix, case: ignore } ]
```
The `not` and `case: ignore` directives can both be applied on the same rule. The following demonstrates the `suffix` directive which specifies that `X-Forwarded-For` should have been received from the proxy with a field value not starting with "a" nor "A":
```YAML
- [ X-Forwarded-For, { value: a, not: prefix, case: ignore } ]
```
In addition to HTTP/2 trailer header replay discussed above, Proxy Verifier also supports trailer header verification, as demonstrated below:
```YAML
proxy-response:
# Other verifications...
trailers:
fields:
# Verify the client receives the response trailers.
- [ x-test-trailer-1, { value: one, as: equal } ]
- [ x-test-trailer-2, { value: two, as: equal } ]
```
Per [RFC 9110 section 5.3](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.3),
Proxy Verifier supports repeated ordinary HTTP fields by combining their values
with `", "` in the order they were received before applying the verification
rule. Order therefore matters for repeated field lines that can be legally
combined.
When a verification rule uses a sequence value for an ordinary field, Proxy
Verifier applies the same RFC combination rule to the expected value. The two
examples below are therefore equivalent:
```YAML
server-response:
headers:
fields:
- [Cache-Control, no-store]
- [Cache-Control, max-age=0]
proxy-response:
headers:
fields:
- [Cache-Control, { value: [no-store, max-age=0], as: equal }]
```
```YAML
server-response:
headers:
fields:
- [Vary, "accept-encoding, accept-language"]
proxy-response:
headers:
fields:
- [Vary, { value: [accept-encoding, accept-language], as: equal }]
```
`Set-Cookie` is the exception called out by
[RFC 9110 section 5.3](https://www.rfc-editor.org/rfc/rfc9110.html#section-5.3).
Because `Set-Cookie` field lines are not combined into a comma-separated list,
Proxy Verifier provides a separate `set-cookie-verifications` node for them.
Positive entries in `set-cookie-verifications` are matched against the
received `Set-Cookie` lines without requiring a specific order. Each received
`Set-Cookie` line can satisfy at most one positive entry. Any additional
received `Set-Cookie` lines after the listed positive rules are ignored.
Negative entries such as `{ as: absent }`, `{ not: equal }`, or
`{ not: contains }` do not consume a received `Set-Cookie` line. They assert
that no received `Set-Cookie` line matches that rule. Within
`set-cookie-verifications`, a `value` array expands to multiple independent
checks of the same rule. For `Set-Cookie`, `{ as: absent }` without a `value`
asserts that no `Set-Cookie` field lines are present at all, while
`{ as: absent, value: ... }` asserts that no received `Set-Cookie` line
contains that value. The legacy `fields` syntax still supports
`- [Set-Cookie, { as: absent }]` for the no-`Set-Cookie` case.
```YAML
# Example: positive Set-Cookie checks match individual cookie lines in any
# order, one received line per positive rule.
server-response:
headers:
fields:
- [:status, 200]
- [Content-Type, text/html]
- [Content-Length, '11']
- [Set-Cookie, "A1=111"]
- [Set-Cookie, "B1=333"]
- [Set-Cookie, "C1=555"]
proxy-response:
headers:
set-cookie-verifications:
- { value: A1=, as: contains }
- { value: B1=333, as: equal }
# Example: a value array expands into multiple independent positive checks.
proxy-response:
headers:
set-cookie-verifications:
- { value: [A1=, A3=, A1S], as: contains }
# Example: absent with a value asserts that no received Set-Cookie line
# contains that value. An array expands into multiple negative checks.
proxy-response:
headers:
set-cookie-verifications:
- { value: [D1=, _ebd], as: absent }
# Example: absent without a value asserts that there are no Set-Cookie lines
# at all in the received message.
proxy-response:
headers:
set-cookie-verifications:
- { as: absent }
# Example: the legacy fields syntax still works for the no-Set-Cookie case.
proxy-response:
headers:
fields:
- [Set-Cookie, { as: absent }]
```
### Request Method Verification
For HTTP/1, request method verification is specified via the scalar `method`
node in `proxy-request`. If present, Proxy Verifier verifies that the proxy
sent that exact HTTP method token.
For example, the following `proxy-request` node directs the Verifier server to
verify that the proxy sends an HTTP/1 `POST` request:
```YAML
proxy-request:
method: POST
```
For HTTP/2 and HTTP/3, method verification is specified via the `:method`
pseudo header field using the normal field verification syntax:
```YAML
proxy-request:
headers:
fields:
- [ :method, { value: POST, as: equal } ]
```
### URL Verification
In a manner similar to field verification described above, a mechanism exists
to verify the parts of URLs in the request line being received from the proxy
by the server. This mechanism is useful for verifying the request targets of
HTTP/1 requests. For HTTP/2 requests, the analogous verification is done via
pseudo header field verification of the `:scheme`, `:authority`, and
`:path` fields using the above described field verification.
Request target verification rules are stipulated via a sequence value for the
`url` node rather than the scalar value used in `client-request` nodes. The
verifiable parts of the request target follow the URI specification (see
[RFC 3986 section 3](https://tools.ietf.org/html/rfc3986#section-3) for the formal
definition of these various terms):
* `scheme`
* `host`
* `port`
* `authority` (also known as `net-loc`, the combination of `host` and `port`),
* `path`
* `query`
* `fragment`
For example, consider the following request line:
```
GET http://example.one:8080/path?query=q#Frag HTTP/1.1
```
The request URL in this case is case is
`http://example.one:8080/path?query=q#Frag`. The Verifier server can be
configured to verify the various parts of such URLs using the same map sytax
explained above for field verification using any of those same directives
(`equal`, `contains`, etc.). Continuing with this example URL, the following
uses the `equal` directive for each part of the URL, thus verifying that the
request URL exactly matches the URL given in this example and emitting a
verification error message if any parts of the URL do not match:
```YAML
proxy-request:
url:
- [ scheme, { value: http, as: equal } ]
- [ host, { value: example.one, as: equal } ]
- [ port, { value: 8080, as: equal } ]
- [ path, { value: /path, as: equal } ]
- [ query, { value: query=q, as: equal } ]
- [ fragment, { value: Frag, as: equal } ]
```
Alternatively to `host` and `port`, `authority`, with an alias of `net-loc`, is
supported, which is the combination of the two:
```YAML
- [ authority, { value: example.one:8080, as: equal } ]
```
As another example using other directives, consider a request URL of
`/path/x/y?query=q#Frag`. Verification of this can be specified with the
following:
```YAML
proxy-request:
url:
- [ scheme, { value: http, as: absent } ]
- [ authority, { value: foo, as: absent } ]
- [ path, { value: /path/x/y, as: equal } ]
- [ query, { value: query=q, as: equal } ]
- [ fragment, { value: foo, as: present } ]
```
Note that `scheme` and `authority` parts both use `absent` directives because
this particular URL just has path, query, and fragment components. Thus, with
this verification specification, if the proxy includes scheme or authority in
the request target, it will result in a verification failure. The `path` and
`query` components must match `/path/x/y` and `query=q` exactly because they
use the `equal` directive. The `fragment` of the URL is verified with the
`present` directive in this case, indicating that the received URL from the
proxy only needs to have some fragment of any value to pass this specified
verification.
### Status Verification
Proxy HTTP response status verification is specified in `proxy-response`
nodes. In this case, the response status that the Verifier client should expect
from the proxy is specified in the same way that directs the Verifier server
in what response status should be sent for a given request. Both response code,
such as "403", and HTTP/1 response reason string, such as "Forbidden", are
supported for verification.
For example, the following complete `proxy-response` node directs the Proxy
Verifier client to verify that the proxy replies to the associated HTTP request
with a `404` status:
```YAML
proxy-response:
status: 404
reason: Not Found
```
This verification directive applies to HTTP/1 transactions. For HTTP/2, status
verification is specified via `:status` pseudo header field verification using
the field verification mechanism described above.
### Body Verification
In a manner similar to field verification described above, a mechanism exists
to verify the body content of a request or response. To specify a rule for
verifying body content, a new `verify` node should be added under the `content`
node. The rules follow the same map syntax as described for field verification.
```YAML
proxy-request:
content:
verify: {value: test1, as: equal}
proxy-response:
content:
verify: {value: test2, as: contains}
```
The `value` node in the `verify` node can be ommited if the `data` node is used
to specify the content, since body content can get very long, and/or multi-lined.
However, `value` node has priority over the `data` node, meaning if there is a
`value` node, then the `data` will be ignored.
```YAML
proxy-request:
content:
encoding: plain
data: test1
verify: {as: equal}
proxy-response:
content:
encoding: plain
data: test2
verify: {as: contains}
```
Note that only one body verification node is needed, even if multiple `DATA` frames
are specified as described in [HEADERS and DATA frame](#headers-and-data-frame). The
verification node need to combine all the values specified for the `DATA` frames.
See the example below:
```YAML
client-request:
frames:
- HEADERS:
headers:
fields:
- [:method, POST]
- [:scheme, https]
- [:authority, example.data.com]
- [:path, /a/path]
- [Content-Type, text/html]
- [uuid, 1]
- DATA:
content:
encoding: plain
data: client_data_1
- DATA:
content:
encoding: plain
data: client_data_2
proxy-request:
content:
encoding: plain
data: client_data_1client_data_2
verify: {as: equal}
```
### Example Replay File
The sections leading up to this one have described each of the major components
of a YAML Proxy Verifier replay file. Putting these components together, the
following complete sample replay file demonstrates the description of the
replay of two sessions: one an HTTP/1.1 session, the second an HTTP/2 session.
Each session contains a single transaction with verification of certain parts
of the HTTP messages.
```YAML
meta:
version: '1.0'
sessions:
#
# First session: since there is no "protocol" node for this session,
# HTTP/1.1 over TCP (no TLS) is assumed.
#
- transactions:
#
# Direct the Proxy Verifier client to send a POST request with a body of
# 399 bytes.
#
- client-request:
method: POST
url: /pictures/flower.jpeg
version: '1.1'
headers:
fields:
- [ Host, www.example.com ]
- [ Content-Type, image/jpeg ]
- [ Content-Length, '399' ]
- [ uuid, first-request ]
# A "content" node is not needed if a Content-Length field is specified.
#
# Direct the Proxy Verifier server to verify that the request received from
# the proxy has a path in the request target that contains "flower.jpeg",
# has a path that is not prefixed with "JPEG" (case insensitively),
# and has the Content-Length field of any value.
#
proxy-request:
url:
- [ path, { value: flower.jpeg, as: contains } ]
- [ path, { value: JPEG, not: prefix, case: ignore } ]
headers:
fields:
- [ Content-Length, { value: '399', as: present } ]
#
# Direct the Proxy Verifier server to reply with a 200 OK response with a body
# of 3,432 bytes.
#
server-response:
status: 200
reason: OK
headers:
fields:
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
- [ Transfer-Encoding, chunked ]
- [ Connection, keep-alive ]
# Unlike the request which contains a Content-Length, this response
# will require a "content" node to specify the size of the body.
# Otherwise Proxy Verifier has no way of knowing how large the response
# should be.
content:
size: 3432
#
# Direct the Proxy Verifier client to verify that it receives a 200 OK from
# the proxy with a `Transfer-Encoding: chunked` header field.
#
proxy-response:
status: 200
headers:
fields:
- [ Transfer-Encoding, { value: chunked, as: equal } ]
#
# For the second session, we use a protocol node to configure HTTP/2 using an
# SNI of # test_sni in the TLS handshake.
#
- protocol:
stack: http2
tls:
sni: test_sni
transactions:
#
# Direct the Proxy Verifier client to send a POST request with a body of
# 399 bytes.
#
- client-request:
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, /pictures/flower.jpeg ]
- [ Content-Type, image/jpeg ]
- [ uuid, second-request ]
content:
size: 399
#
# Direct the Proxy Verifier server to verify that the request received from
# the proxy has a path pseudo header field that contains "flower.jpeg"
# and has a field "Content-Type: image/jpeg".
#
proxy-request:
url:
- [ path, { value: flower.jpeg, as: contains } ]
headers:
fields:
- [ :method, POST ]
- [ :scheme, https ]
- [ :authority, www.example.com ]
- [ :path, { value: flower.jpeg, as: contains } ]
- [ Content-Type, { value: image/jpeg, as: equal } ]
#
# Direct the Proxy Verifier server to reply with a 200 OK response with a body
# of 3,432 bytes.
#
server-response:
headers:
fields:
- [ :status, 200 ]
- [ Date, "Sat, 16 Mar 2019 03:11:36 GMT" ]
- [ Content-Type, image/jpeg ]
content:
size: 3432
#
# Direct the Proxy Verifier client to verify that it receives a 200 OK from
# the proxy.
#
proxy-response:
status: 200
#
# For the third session, we demonstrate how body verification should be specified.
#
- protocol:
stack: https
tls:
sni: test_sni
transactions:
#
# Direct the Proxy Verifier client to send a POST request with a body of
# 11 bytes.
#
- client-request:
method: POST
url: /a/path
version: '1.1'
headers:
fields:
- [ Host, example.data.com ]
- [ Content-Type, text/html ]
- [ Content-Length, '11' ]
- [ uuid, third-request ]
content:
encoding: plain
data: client_test
size: 11
#
# Direct the Proxy Verifier server to verify that the request received from
# the proxy has body content "client_test".
#
proxy-request:
content:
verify: { value: client_test, as: equal }
#
# Direct the Proxy Verifier server to reply with a 200 OK response with a body
# of 11 bytes.
#
server-response:
status: 200
reason: OK
headers:
fields:
- [ Content-Type, text/html ]
- [ Content-Length, '11' ]
content:
encoding: plain
data: |-
server
test
size: 11
#
# Direct the Proxy Verifier client to verify that the response received from
# the proxy has body content "server\ntest".
#
proxy-response:
content:
encoding: plain
data: |-
server
test
verify: { as: equal }
```
## Installing
### Prebuilt Binaries
Statically linked binaries for Linux and Mac are provided with the release in
the [Releases](https://github.com/yahoo/proxy-verifier/releases) page. If you do
not need your own customized build of Proxy Verifier, the easiest way to start
using it is to simply download the proxy-verifier `tar.gz` for the desired
release, untar it on the desired box, and copy the `verifier-client` and
`verifier-server` binaries to a convenient location from which to run them.
On Linux, these release binaries are built as statically linked musl
executables. They do not depend on glibc or `libnss_*` shared libraries at
runtime, although hostname resolution still consults normal system files such
as `/etc/hosts` and `/etc/resolv.conf`.
### Building from Source
These instructions describe how to build a copy of the Proxy Verifier project
on your local machine for development and testing purposes.
#### Prerequisites
Proxy Verifier is built with CMake and the checked-in
[`CMakePresets.json`](https://github.com/yahoo/proxy-verifier/tree/master/CMakePresets.json)
presets. The build requires:
* CMake 4.2.3 or newer
* Ninja
* a C and C++ compiler
* git
* pkg-config
* OpenSSL 3.5 or newer development files
* nghttp2 1.60 or newer development files
* nghttp3 0.8 or newer development files
* liburing 2.0 or newer development files on Linux (optional)
* jemalloc development files on Linux (optional, recommended)
CMake fetches pinned `libswoc` and `yaml-cpp` sources during configure. HTTP/3
uses OpenSSL 3.5 or newer for the native QUIC transport and nghttp3 for HTTP/3
framing and QPACK. HTTP/2 uses nghttp2. Install these libraries through the
system package manager before configuring Proxy Verifier. ngtcp2 is not
required because OpenSSL provides the QUIC transport. On Linux, CMake enables
io_uring socket readiness support when liburing is installed. Builds without
liburing, non-Linux builds, and kernels that cannot initialize io_uring retain
the existing `poll(2)` behavior. Dynamic Linux builds that enable io_uring
support require the liburing shared library at runtime; portable Linux release
binaries link it statically.
On Linux, CMake also links jemalloc as the memory allocator when pkg-config
finds it, which cuts replay file parsing CPU by about a third: parsing builds
and frees a yaml-cpp node for every element of every replay file, and glibc
malloc spends much of that time contending and consolidating. Static builds
use jemalloc only when its static archive is installed. Configure with
`-DPV_ENABLE_JEMALLOC=OFF` to keep the system allocator. Replay files are
parsed on one thread per hardware thread.
For Mac builds, the following brew command can be helpful:
```bash
brew install \
git \
cmake \
ninja \
nghttp2 \
nghttp3 \
openssl@3 \
pkg-config \
python@3.14
```
#### Build
Configure and build with the `dev` preset:
```
cmake --preset dev
cmake --build --preset dev --parallel
```
This places the build-tree binaries under `build/dev/bin`.
#### Development Docker Images
The Dockerfiles under `docker/alpine_3.24`, `docker/fedora_44`, and
`docker/ubuntu_26.04` are build and development environments, not deployment
images. Each installs OpenSSL, nghttp2, nghttp3, and liburing from its
distribution along with the Proxy Verifier build, formatting, license-audit,
and Uranium/pytest toolchains.
GitHub Actions uses only the Ubuntu 26.04 image. Alpine 3.24 is used to build
the portable Linux release binaries, while Fedora 44 provides an additional
current-distribution development environment. Users who prefer Alpine or
Fedora can build those Dockerfiles for their local build environment.
For deployment, using the statically linked binaries associated with the GitHub
releases is the easiest path for most people.
Tooling for the dev image lives in the `/opt/pv-venv` virtual environment and
is added to the `PATH` by the Dockerfile `ENV` configuration.
See [`tools/CI-IMAGES.md`](tools/CI-IMAGES.md) for details about automated
Ubuntu image publication to GitHub Container Registry and instructions to
build and verify Alpine and Fedora images locally. Build the images from the
repository root. For example:
```bash
docker build -f docker/ubuntu_26.04/Dockerfile -t pv-dev:ubuntu26.04 .
docker run --rm -it -v "$(git rev-parse --show-toplevel)":/workspace \
pv-dev:ubuntu26.04
# Now, within the container.
cd /workspace
cmake --preset dev
cmake --build --preset dev --parallel
ctest --preset dev
./build/dev/urtest.sh --sandbox /tmp/pv-urtest -n "$(nproc)"
```
#### Install
The install root is fully configurable through standard CMake install
variables. The development preset defaults to `/usr/local`. To install the
`dev` build there:
```
cmake --install build/dev
```
This places `verifier-client` and `verifier-server` under `/usr/local/bin` by
default.
#### ASan Instrumentation
Use the `dev-asan` preset for AddressSanitizer instrumentation:
```
cmake --preset dev-asan
cmake --build --preset dev-asan --parallel
```
#### Debug Build
The `dev` and `dev-asan` presets default to `CMAKE_BUILD_TYPE=Debug`.
#### Portable Build
The `portable` preset is used to build release artifacts. It statically links
third-party dependencies from system development packages and automatically
maps the current platform to a release directory name such as `linux-amd64`,
`linux-arm64`, or `darwin-arm64`.
For release builds, it is recommended that you use
`tools/build-portable-binaries`:
```bash
./tools/build-portable-binaries
```
The equivalent manual commands are:
```
cmake --preset portable
cmake --build --preset portable --parallel
cmake --install build/portable --strip
```
The install step stages stripped binaries under
`/tmp/proxy-verifier-v/`.
`--strip` is used to shrink the binaries for optimal distribution. The binary
artifacts in each of the Proxy Verifier release are stripped for this reason.
This makes debugging issues much harder, of course, since symbols are removed
from the binaries. Omit `--strip` if size is less of an issue for you.
On Linux, build portable release artifacts in the `docker/alpine_3.24` image so
the resulting binaries are static musl executables without a glibc or
`libnss_*` runtime dependency. DNS still consults `/etc/hosts` and
`/etc/resolv.conf`. On macOS, the portable build links Proxy Verifier and its
third-party dependencies via static archives while leaving the system libraries
dynamic. After the install step, the staged release binaries land under
`/tmp/proxy-verifier-v/`.
For Linux portable release artifacts on x86_64, the portable build pins
`-march=x86-64 -mtune=generic` for Proxy Verifier. Alpine supplies its system
libraries for the distribution's generic architecture baseline.
The portable presets explicitly use `-O2 -DNDEBUG` rather than the toolchain's
default `Release` optimization level so the staged binaries stay conservative
for portable deployment.
The `` portion comes from the top-level
`project(ProxyVerifier VERSION ...)` declaration in
[`CMakeLists.txt`](https://github.com/yahoo/proxy-verifier/blob/master/CMakeLists.txt).
CMake exposes that value as `PROJECT_VERSION`, and the release layout logic
uses it when `PV_RELEASE_LAYOUT=ON` to turn the default `/tmp` install prefix
into `/tmp/proxy-verifier-v${PROJECT_VERSION}`. The `--version` output uses
that same `PROJECT_VERSION` value via the build's compile definitions, so the
CLI version string and release staging directory stay in sync.
#### Native Build
Use the `native` preset for a host-tuned dynamic build:
```
cmake --preset native
cmake --build --preset native --parallel
```
These builds are intended for running optimally on the build machine rather than
for redistribution. They keep dynamic linking and use `-O3 -DNDEBUG
-march=native -mtune=native` for host-tuned performance.
### Running the Tests
#### Unit Tests
Build the test executable with CMake, then run it through CTest:
```
cmake --preset dev
cmake --build --preset dev --parallel
ctest --preset dev
```
#### Uranium Tests
Proxy Verifier's end-to-end tests use the pytest-based Uranium framework.
Tests live under `tests/uranium_tests` as ordinary `test_*.py` modules; replay
YAML, JSON, and gold files in those directories are test data. After
configuring a preset, CMake generates an `urtest.sh` wrapper in that build
directory. To run the tests for `dev`, use:
```
./build/dev/urtest.sh
```
That wrapper selects the `verifier-client` and `verifier-server` binaries from
its build tree and creates an isolated sandbox for each test case. Passed-case
sandboxes are removed, while failed-case sandboxes are retained for diagnosis.
It also enables `cdifflib` to accelerate unified-diff diagnostics for failed
gold-file comparisons; the framework falls back to Python's standard-library
matcher if the optional extension is unavailable.
Use pytest's `-k` option to run a matching subset. For example, this runs the
HTTP and HTTPS tests:
```
./build/dev/urtest.sh -k 'http or https'
```
Use pytest-xdist's `-n` option for parallel runs. Worker processes use separate
sandboxes and a shared port allocator:
```
./build/dev/urtest.sh -n 8 -k 'http or https'
```
Register process-output expectations with the explicit stream API. Regex
expectations require an explanation so failures report the intended behavior:
```python
server.stdout.contains("Ready with 3 transactions",
"The server should parse all three transactions.")
server.stdout.excludes("Violation:",
"The server should not report verification errors.")
server.stdout.matches_gold("gold/server.gold")
client.expect_return_codes(0, 1)
```
The `stdout`, `stderr`, and `return_codes` properties are read-only; assignment
and augmented assignment are not expectation-registration APIs. Use
`server.stdout.reset()` when earlier stream expectations must be discarded
explicitly.
The wrapper accepts normal pytest options; run `./build/dev/urtest.sh --help`
for the complete list. Use `--sandbox PATH` to select the sandbox root. CI
sharding is also supported with the zero-based `SHARD` and `SHARDCNT`
environment variables.
## Usage
This section describes how to run the Proxy Verifier client and server at
the command line.
### Required Arguments
At a high level, Proxy Verifier is run in the following manner:
1. Run the verifier-server with the set of HTTP and HTTPS ports to listen on
configured though the command line. The directory containing the replay file
is also configured through a command line argument.
1. Configure and run the proxy to listen on a set of HTTP and HTTPS ports and
to proxy those connections to the listening verifier-server ports.
1. Run the verifier-client with the sets of HTTP and HTTPS ports on which to
connect configured though the command line. The directory containing the
replay file is also configured through a command line argument.
Here's an example invocation of the verifier-server, configuring it to listen on
localhost port 8080 for HTTP connections and localhost port 4443 for HTTPS
connections:
```
verifier-server \
run \
--listen-http 127.0.0.1:8080 \
--listen-https 127.0.0.1:4443 \
--server-cert \
--ca-certs \
```
Here's an example invocation of the verifier-client, configuring it to connect to
the proxy which has been configured to listen on localhost port 8081 for HTTP
connections and localhost port 4444 for HTTPS connections:
```
verifier-client \
run \
--client-cert \
--ca-certs \
--connect-http 127.0.0.1:8081 \
--connect-https 127.0.0.1:4444 \
```
With these two invocations, the verifier-client and verifier-server will replay the
sessions and transactions in `` and perform any field
verification described therein.
On the server either `--listen-http` or `--listen-https` or both must be
provided. That is, for example, if you are only testing HTTPS traffic, you may
only specify `--listen-https`. The same is true on the client: either
`--connect-http` or `--connect-https` or both must be provided. These address
arguments take a comma-separated list of address/port pairs to specify multiple
listening or connecting sockets. The processing of these arguments
automatically detects any IPv6 addresses if provided. The client's processing
of `--connect-http` and `--connect-https` arguments will resolve fully
qualified domain names.
Note that the `--client-cert` and `--server-cert` both take either a
certificate file containing the public and private key or a directory
containing pem and key files. Similarly, the `--ca-certs` takes either a file
containing one or more certificates or a directory with separate certificate
files. For convenience, the
[tests/keys](https://github.com/bneradt/proxy-verifier/tree/expand_readme/tests/keys)
directory contains key files which can be used for testing. These certificate
arguments are only required if HTTPS traffic will be replayed.
### Optional Arguments
#### --format \
Each transaction has to be uniquely identifiable by the client and server in a
way that is consistent across both replay file parsing and traffic replay
processing. Whatever attributes we use from the messages to uniquely identify
transactions is called the "key" for the dataset. The ability to uniquely
identify these messages is important for at least the following reasons:
* When the Verifier server receives a request, it has to know from which of the
set of parsed transactions it should generate a response. At the time of
processing an incoming message, all it has to go on is the request header
line and the request header fields. From these, it has to be able to identify
which of the potentially thousands of parsed transactions from the replay input
files it should generate a response.
* When the client and server perform field verification, they need to know what
particular verification rules specified in the replay files should be applied
to the given incoming message.
* If the client and server are processing many transactions, generic log
messages could be near useless if there was not a way for the logs to
identify individual transactions to the user somehow.
By default the Verifier client and server both expect a `uuid` header field
value to function as the key.
If the user would like to use other attributes as a key, they can specify
something else via the `--format` argument. The format argument currently
supports generating a key on arbitrary field values and the `URL` of the
request. Some example `--format` expressions include:
* `--format "{field.uuid}"`: This is the default key format. It treats the UUID
header field value as the transaction key.
* `--format "{url}"`: Treat the request `URL` as the key.
* `--format "{field.host}"`: Treat the `Host` header field value as the key.
* `--format "{field.host}/{url}"`: Treat the combination of the `Host` header
field and the request `URL` as the key.
#### --keys \
`--keys` can be passed to the verifier-client to specify a subset of keys from
the replay file to run. Only the transactions from the space-separated list of
keys will be replayed. For example, the following invocation will only run the
transactions with keys whose values are 3 and 5:
```
verifier-client \
run \
\
127.0.0.1:8082 \
127.0.0.1:4443 \
--keys 3 5
```
This is a client-side only option.
#### --verbose
Proxy Verifier has four levels of verbosity that it can run with:
| Verbosity | Description |
| --------- | ----------- |
| error | Transactions either failed to run or failed verification. |
| warning | A non-failing problem occurred but something is likely to go wrong in the future. |
| info | High level test execution information. |
| diag | Low level debug information. |
Each level implies the ones above it. Thus, if a user specifies a verbosity
level of `warning`, then both warning and error messages are reported.
By default, Proxy Verifier runs at `info` verbosity, only producing summary
output by both the client and the server along with any warnings and errors it
found. This can be tweaked via the `--verbose` flag. Here's an example of requesting
the most verbose level of logging (`diag`):
```
verifier-client \
run \
\
127.0.0.1:8082 \
127.0.0.1:4443 \
--verbose diag
```
#### --interface \
Initiate connections from the specified interface, such as eth0:1.
This is a client-side only option.
#### --no-proxy
As explained above, replay files contain traffic information for both client to
proxy traffic and proxy to server traffic. Under certain circumstances it may
be helpful to run the Verifier client directly against the Verifier server.
This can be useful while developing Proxy Verifier itself, for example,
allowing the developer to do some limited testing without requiring the setup
of a test proxy.
To support this, the Verifier client has the `--no-proxy` option. If this
option is used, then the client has its expectations configured such that it
assumes it is run against the Verifier server rather than a proxy. Effectively
this means that instead of trying to run the client to proxy traffic, it will
instead act as the proxy host for the Verifier server and will run the proxy to
server traffic. Concretely, this means that the Verifier client will replay the
`proxy-request` and `proxy-response` nodes rather than the `client-request` and
`client-response` nodes.
This is a client-side only option.
#### --strict
Generally, very little about the replayed traffic is verified except what is
explicitly specified via field verification (see above). This is by design,
allowing the user to replay traffic with only the requested content being
verified. In high-volume cases, such as situations where Proxy Verifier is
being used to scale test the proxy, traffic verification may be considered
unimportant or even unnecessarily noisy. If, however, the user wants every
field to be verified regardless of specification, then the `--strict` option
can be passed to either or both the Proxy Verifier client and server to report
any verification issues against every field specified in the replay file.
#### --allow-unprocessed-verifications
By default, Proxy Verifier fails if a transaction had configured verification
rules but the corresponding message never arrived, so the verification logic
was never reached. On the verifier-server this is checked at shutdown for
unprocessed proxy-request verification. On the verifier-client this is checked
after replay completes for unprocessed proxy-response verification. This
catches cases such as the proxy crashing or otherwise failing to forward
traffic.
This does not apply to `proxy-request.expect: absent`. For that feature,
"never reached" is the success case and an arriving request fails verification.
It also does not apply to `proxy-request.expect: present`, because that is an
explicit requirement that the keyed request reach the Verifier server.
If a test intentionally tolerates that condition, pass
`--allow-unprocessed-verifications` to the verifier-client or verifier-server
to suppress that failure.
#### --rate \
By default, the client will replay the session and transactions in the replay
files as fast as possible. If the user desires to configure the client to
replay the transactions at a particular rate, they can provide the `--rate`
argument. This argument takes the number of requests per second the client will
attempt to send requests at.
Note session and transaction timing data can be specified in the replay files.
These are provided via `start-time` nodes for each `session` and `transaction`.
`start-time` takes as a value the number of nanoseconds since Unix epoch (or
whatever other time of reference observed by all `start-time` nodes in the set
of replay files being run) associated for that session or transaction. With
this timing information, if `--rate` is provided, Proxy Verifier simply scales
the relative time deltas between sessions and transactions that appropriately
achieves the desired transaction rate. [Traffic
Dump](https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/traffic_dump.en.html)
records such timing information when it writes replay files. In the absence of
`start-time` nodes, Proxy Verifier will attempt to apply an appropriate uniform
delay across the sessions and transactions to achieve the specified `--rate`
value.
This is a client-side only option.
#### --repeat \
By default, the client will replay all the transactions once in the set of
input replay files. If the user would like the client to automatically repeat
this set a number of times, they can provide the `--repeat` option. The
argument takes the number of times the client should replay the entire dataset.
This is a client-side only option.
#### --run-continuously
Run the set of sessions in the input replay files in an infinite loop. Passing
this option can be thought of as `--repeat` with a value of infinity.
This is a client-side only option.
#### --thread-limit \
Each connection, corresponding to a `session` in a replay file, is dispatched
on the client in parallel. Likewise, each accepted connection on the server is
handled in parallel. Each of these sessions is handled via a single thread of
execution. By default, Proxy Verifier limits the number of threads for handling
these connections to 2,000. This limit can be changed via the `--thread-limit`
option. Setting a value of 1 on the client will effectively cause sessions
to be replayed in serial.
On the server, a thread is needed only while a connection has work to do. An
idle keep-alive HTTP/1 or HTTP/2 connection, such as a proxy's pooled origin
connection waiting for its next request, is set aside without holding a thread
and is handed back to one as soon as more data arrives. The server's
`--thread-limit` therefore bounds the number of connections being actively
served, not the number of open connections. HTTP/3 connections keep their
thread for their whole lifetime.
#### --qlog-dir \
The `--qlog-dir` option remains accepted and Proxy Verifier creates the
specified directory. OpenSSL's native QUIC API does not currently expose qlog
events, however, so the OpenSSL 3.5 implementation reports a diagnostic and
does not write qlog files. qlog diagnostic logging remains disabled by default.
#### --tls-secrets-log-file \
To facilitate debugging, Proxy Verifier supports logging TLS keys for encrypted
replayed traffic. If this option is used, TLS key logging will be appended to
the specified filename. This file can then be provided to protocol analyzers
such as Wireshark to decrypt the traffic. TLS key logging is disabled by
default.
#### --send-buffer-size \
Configure a `SO_SNDBUF` value to set on the server socket. This can be a
helpful parameter to tune when dealing with the replay of large response
bodies. If this optional value is not set, then the `SO_SNDBUF` option is not
set on the socket and, thus, the system default socket buffer size will be
used.
This is a server-side only option.
#### --poll-timeout \
When Proxy Verifier performs read and write operations, it does so using
non-blocking sockets with a timeout. By default, this timeout is set to 5
seconds (5,000 milliseconds). This optional argument provides a way to specify a
different timeout in milliseconds for these operations.
#### --io-uring \
On Linux builds with liburing, Proxy Verifier uses one process-wide io_uring
to multiplex socket readiness waits. This avoids creating an io_uring for each
connection thread. The HTTP, TLS, and replay state machines remain unchanged,
and Proxy Verifier transparently uses `poll(2)` when io_uring is unavailable.
The mode can be one of:
* `auto` uses io_uring when both the build and running kernel support it, then
falls back to `poll(2)` if initialization is unavailable. This is the
default.
* `off` always uses `poll(2)`. This is useful as the control side of a
production performance comparison using the same binary.
* `required` exits with an error unless io_uring initializes successfully and
does not fall back if the backend later fails. Use this for an io_uring
production canary so kernel security policy or a container seccomp profile
cannot silently turn the test into a `poll(2)` run.
Regardless of the selected mode, zero or negative timeouts use a direct
`poll(2)` call because they do not benefit from asynchronous dispatch.
Set the CMake option `PV_ENABLE_IO_URING=OFF` to omit io_uring support even if
liburing is installed.
## Tools
This section describes how to use some of the scripts under the [tools](tools) directory.
### Replay Gen [replay-gen.py](tools/replay-gen.py)
This tool is used to generate mock replay files for easy testing.
Listed below are the available arguments for this script.
#### -n,--number \
Number of total transactions.
#### -tl,--trans-lower \
The lower limit of transactions per session.
#### -tu,--trans-upper \
The upper limit of transactions per session.
#### -sl,--sess-lower \
The lower limit of sessions per file.
#### -su,--sess-upper \
The upper limit of sessions per file.
#### -tp,--trans-protocols \
A comma separated list of protocols that are allowed to be generated.
Available options are: http, tls, h2, all.
#### -u,--url-file \
Path to a file with the list of URLs that can be used.
The URL list file can be acquired by running the [Remap Config to URL List]() script described below.
#### -o,--output \
Path to a directory where the replay files are generated.
#### -p,--prefix \
Prefix for the replay file names.
#### -j,--out-json
Dump replay files in JSON format. By default replay files will be formatted as YAML.
### Remap Config to URL List [remap-config-to-url-list.py](tools/remap-config-to-url-list.py)
This tool converts a `remap.config` file to a URL list file that is used by [Replay Gen](#replay-gen-replay_genpytoolsreplay_genpy)
Listed below are the available arguments for this script.
#### -o,--output \
A filename to which to write the list of URLs. Defaults to `stdout`.
#### --no-ip
Ignore ip address (in the "replacement" section) in the `remap.config` file.
## Contribute
Please refer to [CONTRIBUTING](CONTRIBUTING.md) for information about how to get involved. We welcome issues, questions, and pull requests.
## License
This project is licensed under the terms of the Apache 2.0 open source license.
Please refer to [LICENSE](LICENSE) for the full terms.