The build was uneventful. It's screwing and slotting things into places they're supposed to be.
(As an aside; this struck me to be quite true, as well: https://twitter.com/whitequark/status/986217575569735680 )
Burn-in proceeded in the usual phases: cpu and memory, then disks, with a focus on power consumption.
Initially I let memtest-x86 run for a day or so just in case anything was horribly broken.
Then, booting into the excellent system-rescue-cd, I ran a combination of stress, primes, pi and bonnie for a while. I was also curious to see the values from the kill-a-watt, since a property of this build was to lower power consumption from the previous server. It broke out like this:
-
idle-idle (no drives): 20W
-
memtest86 (no drives): 30W
-
sysrescuecd (5 drives, no load): 46W
-
stress --cpu 4 : 75W
-
stress --cpu 8, pi --cpu 8 @ load of 16: 75W
-
stress + pi + bonnie: 85W
-
stress --cpu 8 --io 4 --vm 4
-
sysstress-cli 8 threads, 32M digits,
-
bonnie++ -s 32g -n 512:1k:4g:100
I've never really monitored or ensured that disk activity was near "maximums", and in the last few years for no particular reason, I was interested to make sure that this was true early on in the burn-in. Plus, I figured it would function as a good way to throw a lot of activity/load at the disks to see if they would fail early before I started relying on them.
Bonnie++ was useful, but the real tool to use here is fio:
/dev/sda
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
randread: (g=1): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
seqread: (g=2): rw=read, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
seqwrite: (g=3): rw=write, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
randmix: (g=4): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
seqmix: (g=5): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.0.9
Starting 6 processes
randmix: Laying out IO file(s) (1 file(s) / 1024MB)
seqmix: Laying out IO file(s) (1 file(s) / 1024MB)
randwrite: (groupid=0, jobs=1): err= 0: pid=28702: Sat Mar 17 04:34:56 2018
write: io=1024.0MB, bw=1666.6KB/s, iops=416 , runt=629182msec
slat (usec): min=2 , max=223550 , avg=87.24, stdev=2098.82
clat (usec): min=970 , max=858733 , avg=153511.58, stdev=77298.22
lat (msec): min=1 , max=858 , avg=153.60, stdev=77.39
clat percentiles (msec):
| 1.00th=[ 21], 5.00th=[ 68], 10.00th=[ 77], 20.00th=[ 95],
| 30.00th=[ 112], 40.00th=[ 126], 50.00th=[ 141], 60.00th=[ 155],
| 70.00th=[ 174], 80.00th=[ 196], 90.00th=[ 243], 95.00th=[ 306],
| 99.00th=[ 437], 99.50th=[ 486], 99.90th=[ 570], 99.95th=[ 627],
| 99.99th=[ 685]
bw (KB/s) : min= 265, max= 3419, per=100.00%, avg=1681.68, stdev=366.95
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.09%, 20=0.86%, 50=1.33%
lat (msec) : 100=20.94%, 250=67.68%, 500=8.70%, 750=0.39%, 1000=0.01%
cpu : usr=0.75%, sys=2.29%, ctx=257413, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=0/w=0/d=262144, short=r=0/w=0/d=0
randread: (groupid=1, jobs=1): err= 0: pid=29173: Sat Mar 17 04:34:56 2018
read : io=1024.0MB, bw=2121.1KB/s, iops=530 , runt=494173msec
slat (usec): min=2 , max=431 , avg=34.47, stdev=12.83
clat (usec): min=641 , max=1257.7K, avg=120602.90, stdev=81188.62
lat (usec): min=645 , max=1257.8K, avg=120638.47, stdev=81188.53
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 28], 10.00th=[ 39], 20.00th=[ 58],
| 30.00th=[ 75], 40.00th=[ 90], 50.00th=[ 104], 60.00th=[ 120],
| 70.00th=[ 139], 80.00th=[ 169], 90.00th=[ 223], 95.00th=[ 277],
| 99.00th=[ 408], 99.50th=[ 469], 99.90th=[ 611], 99.95th=[ 668],
| 99.99th=[ 840]
bw (KB/s) : min= 815, max= 4063, per=100.00%, avg=2121.73, stdev=226.86
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.27%, 20=2.23%, 50=13.43%
lat (msec) : 100=31.15%, 250=46.04%, 500=6.53%, 750=0.31%, 1000=0.02%
lat (msec) : 2000=0.01%
cpu : usr=0.91%, sys=2.58%, ctx=262450, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0
seqread: (groupid=2, jobs=1): err= 0: pid=29532: Sat Mar 17 04:34:56 2018
read : io=1024.0MB, bw=176647KB/s, iops=44161 , runt= 5936msec
slat (usec): min=1 , max=797 , avg= 5.57, stdev= 8.90
clat (usec): min=172 , max=426325 , avg=1442.31, stdev=3509.15
lat (usec): min=192 , max=426329 , avg=1448.08, stdev=3509.02
clat percentiles (usec):
| 1.00th=[ 478], 5.00th=[ 580], 10.00th=[ 708], 20.00th=[ 860],
| 30.00th=[ 940], 40.00th=[ 988], 50.00th=[ 1032], 60.00th=[ 1112],
| 70.00th=[ 1208], 80.00th=[ 1352], 90.00th=[ 2224], 95.00th=[ 2640],
| 99.00th=[10816], 99.50th=[11072], 99.90th=[18048], 99.95th=[34560],
| 99.99th=[199680]
bw (KB/s) : min=158784, max=184200, per=99.92%, avg=176508.64, stdev=9110.59
lat (usec) : 250=0.01%, 500=1.42%, 750=10.73%, 1000=31.44%
lat (msec) : 2=45.13%, 4=8.72%, 10=0.67%, 20=1.79%, 50=0.05%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%
cpu : usr=7.95%, sys=22.91%, ctx=33402, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=262144/w=0/d=0, short=r=0/w=0/d=0
seqwrite: (groupid=3, jobs=1): err= 0: pid=29538: Sat Mar 17 04:34:56 2018
write: io=1024.0MB, bw=174704KB/s, iops=43676 , runt= 6002msec
slat (usec): min=1 , max=8910 , avg= 7.66, stdev=18.59
clat (usec): min=215 , max=66990 , avg=1456.09, stdev=2359.73
lat (usec): min=219 , max=75611 , avg=1463.97, stdev=2360.57
clat percentiles (usec):
| 1.00th=[ 490], 5.00th=[ 588], 10.00th=[ 596], 20.00th=[ 620],
| 30.00th=[ 788], 40.00th=[ 948], 50.00th=[ 1064], 60.00th=[ 1144],
| 70.00th=[ 1288], 80.00th=[ 1576], 90.00th=[ 2288], 95.00th=[ 2768],
| 99.00th=[11200], 99.50th=[11712], 99.90th=[34560], 99.95th=[37120],
| 99.99th=[67072]
bw (KB/s) : min=154003, max=189544, per=100.00%, avg=176280.18, stdev=9821.98
lat (usec) : 250=0.04%, 500=1.13%, 750=26.85%, 1000=14.71%
lat (msec) : 2=44.63%, 4=10.35%, 10=0.33%, 20=1.69%, 50=0.24%
lat (msec) : 100=0.02%
cpu : usr=10.06%, sys=29.26%, ctx=28879, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=0/w=0/d=262144, short=r=0/w=0/d=0
randmix: (groupid=4, jobs=1): err= 0: pid=29546: Sat Mar 17 04:34:56 2018
read : io=838560KB, bw=1237.4KB/s, iops=309 , runt=677719msec
slat (usec): min=2 , max=298226 , avg=72.39, stdev=1985.39
clat (usec): min=758 , max=949674 , avg=129640.81, stdev=100279.49
lat (usec): min=793 , max=949709 , avg=129714.28, stdev=100350.24
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 26], 10.00th=[ 36], 20.00th=[ 54],
| 30.00th=[ 71], 40.00th=[ 86], 50.00th=[ 102], 60.00th=[ 122],
| 70.00th=[ 147], 80.00th=[ 186], 90.00th=[ 265], 95.00th=[ 347],
| 99.00th=[ 486], 99.50th=[ 529], 99.90th=[ 644], 99.95th=[ 685],
| 99.99th=[ 791]
bw (KB/s) : min= 416, max= 2448, per=100.00%, avg=1242.22, stdev=496.71
write: io=210016KB, bw=317323 B/s, iops=77 , runt=677719msec
slat (usec): min=4 , max=159793 , avg=103.09, stdev=2361.00
clat (usec): min=612 , max=1371.7K, avg=308029.22, stdev=253485.64
lat (usec): min=660 , max=1371.7K, avg=308133.44, stdev=253483.12
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 18], 10.00th=[ 32], 20.00th=[ 60],
| 30.00th=[ 87], 40.00th=[ 153], 50.00th=[ 265], 60.00th=[ 363],
| 70.00th=[ 453], 80.00th=[ 553], 90.00th=[ 676], 95.00th=[ 766],
| 99.00th=[ 930], 99.50th=[ 988], 99.90th=[ 1106], 99.95th=[ 1139],
| 99.99th=[ 1221]
bw (KB/s) : min= 23, max= 752, per=100.00%, avg=310.13, stdev=155.44
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.10%, 10=0.61%, 20=2.72%, 50=14.29%
lat (msec) : 100=27.90%, 250=35.18%, 500=13.47%, 750=4.58%, 1000=1.05%
lat (msec) : 2000=0.08%
cpu : usr=0.69%, sys=2.03%, ctx=258281, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=209640/w=0/d=52504, short=r=0/w=0/d=0
seqmix: (groupid=5, jobs=1): err= 0: pid=30064: Sat Mar 17 04:34:56 2018
read : io=839528KB, bw=11785KB/s, iops=2946 , runt= 71234msec
slat (usec): min=1 , max=69913 , avg=20.52, stdev=312.63
clat (usec): min=118 , max=230218 , avg=20261.42, stdev=32464.50
lat (usec): min=163 , max=230232 , avg=20282.56, stdev=32465.65
clat percentiles (usec):
| 1.00th=[ 458], 5.00th=[ 644], 10.00th=[ 764], 20.00th=[ 932],
| 30.00th=[ 1096], 40.00th=[ 1432], 50.00th=[ 2192], 60.00th=[ 5664],
| 70.00th=[10432], 80.00th=[49408], 90.00th=[76288], 95.00th=[92672],
| 99.00th=[119296], 99.50th=[129536], 99.90th=[154624], 99.95th=[164864],
| 99.99th=[220160]
bw (KB/s) : min= 8806, max=16135, per=100.00%, avg=11801.80, stdev=1352.19
write: io=209048KB, bw=2934.7KB/s, iops=733 , runt= 71234msec
slat (usec): min=2 , max=69808 , avg=29.70, stdev=443.22
clat (usec): min=228 , max=200468 , avg=5715.79, stdev=8921.41
lat (usec): min=268 , max=200490 , avg=5746.14, stdev=8940.66
clat percentiles (usec):
| 1.00th=[ 580], 5.00th=[ 756], 10.00th=[ 860], 20.00th=[ 1012],
| 30.00th=[ 1224], 40.00th=[ 1528], 50.00th=[ 2128], 60.00th=[ 2992],
| 70.00th=[ 5856], 80.00th=[ 9152], 90.00th=[12992], 95.00th=[21888],
| 99.00th=[43264], 99.50th=[53504], 99.90th=[83456], 99.95th=[91648],
| 99.99th=[160768]
bw (KB/s) : min= 2096, max= 4079, per=100.00%, avg=2938.51, stdev=348.91
lat (usec) : 250=0.03%, 500=1.18%, 750=7.28%, 1000=15.20%
lat (msec) : 2=24.52%, 4=10.97%, 10=12.79%, 20=6.50%, 50=5.55%
lat (msec) : 100=13.27%, 250=2.72%
cpu : usr=3.80%, sys=9.68%, ctx=114107, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=209882/w=0/d=52262, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=1024.0MB, aggrb=1666KB/s, minb=1666KB/s, maxb=1666KB/s, mint=629182msec, maxt=629182msec
Run status group 1 (all jobs):
READ: io=1024.0MB, aggrb=2121KB/s, minb=2121KB/s, maxb=2121KB/s, mint=494173msec, maxt=494173msec
Run status group 2 (all jobs):
READ: io=1024.0MB, aggrb=176646KB/s, minb=176646KB/s, maxb=176646KB/s, mint=5936msec, maxt=5936msec
Run status group 3 (all jobs):
WRITE: io=1024.0MB, aggrb=174704KB/s, minb=174704KB/s, maxb=174704KB/s, mint=6002msec, maxt=6002msec
Run status group 4 (all jobs):
READ: io=838560KB, aggrb=1237KB/s, minb=1237KB/s, maxb=1237KB/s, mint=677719msec, maxt=677719msec
WRITE: io=210016KB, aggrb=309KB/s, minb=309KB/s, maxb=309KB/s, mint=677719msec, maxt=677719msec
Run status group 5 (all jobs):
READ: io=839528KB, aggrb=11785KB/s, minb=11785KB/s, maxb=11785KB/s, mint=71234msec, maxt=71234msec
WRITE: io=209048KB, aggrb=2934KB/s, minb=2934KB/s, maxb=2934KB/s, mint=71234msec, maxt=71234msec
Disk stats (read/write):
sda: ios=739698/536984, merge=203565/93548, ticks=60811136/56677146, in_queue=117488702, util=100.00%
After a couple of days of this, I was a/ impatient and b/ confident in the hardware. :) It was time to cut over to a new OS image.