Simple Black Box
- Dieter Plaetinck / Dieter_be
- new in NYC
- Vimeo backend
@Vimeo
- upload server
- clients speak upload protocol over HTTP
- manages files on disk
- manages files in s3, swift (HTTP)
- metrics to statsd (UDP)
in general:
- messages/data i/o, multiple protocols
- changes to filesystem, does logging
- configuration that controls behavior
during development how do I test that >=1 daemon(s) behave correctly?
- treat subject as black box (code agnostic. this is not unit testing)
- send inputs, capture all possible output, assert on that
- assert on what OS tells you about the processes
- test case per scenario
- E_NOTFOUND → DIY
- Simple Black Box:
shellscript tool/framework with plugins for common inputs and probes
A test run...
test_init
set up sandbox (copy dir)
test_pre
- programatically modify configuration
enable/disable settings
set good/faulty params - verify initial state
test_start
- set up probes (internally: tcpdump, ngrep, ..)
set_http_probe swift "host $host_swift"
set_http_probe tag "-d lo host lo and port $port" - launch process, capturing stdout,stderr,logs
- send input to process, http request, upload file, ...
test_while
assert num_procs "$regex" "$num_procs" (pgrep)
assert listening "tcp:8080" (lsof)
test_stop
kill_gracefull "$process"
remove_http_probe swift
remove_http_probe tag
assert num_procs "$regex" 0
assert listening "tcp:8080" 0
test_post
assert object_md5sum $swift_object
assert num_http_req swift 'GET /auth/v1.0' 1 1
assert http_resp swift "^PUT /v1/AUTH HTTP" 202
assert num_statsd_req lo 'upload.put:1|c' 1 1
assert no_errors $output/stderr_* $log (grep)
assert pattern "all is ok" $output/stdout
extra #1
use case:- N processes interacting, exchanging messages and acting upon them
- assertion fails, I want to debug what went wrong
- events are scattered across M logfiles from app+probes
- need central chronological overview of all events
extra #1
→ logstash session to ingest all events during SBB session → logstash as a local debugging tool
- debug events for given testcase (distributed stacktrace)
- compare behavior across test cases
extra #2 (WIP): profile-process
extra #2 (WIP): profile-process
