Both sides previous revisionPrevious revisionNext revision | Previous revision |
linux [2020/04/04 15:01] – [Brother HL-2130] mschubi | linux [2023/06/26 12:02] (current) – [Raspi4 mit Geekworm X862 V2.0 und BAITITON M.2 NGFF 128GB SSD] mschubi |
---|
====== Linux ====== | ====== Linux ====== |
| |
| ====== Plattendurchsatz anzeigen ====== |
| |
| <code bash> |
| # aller 20 Sekunden für sdb und sdc mit Ausgabe in MB/s |
| iostat -d 20 -p sdb sdc -m |
| </code> |
| |
| ===== Raspi4 mit Geekworm X862 V2.0 und BAITITON M.2 NGFF 128GB SSD ===== |
| |
| Ich habe zwei Raspi 4 für den ESXi Fling mit VSAN laufen. Der ganze USB/SD Card Kram hat den Nachteil, dass die Karten/Sticks langsam sind und nicht lange halten. Deshalb habe ich mir zwei unterschiedliche Adapter für den Raspi4 zugelegt. |
| Einmal einen Geekworm X825 V2.0 für 2,5" SATA Disks und einen Geekworm X862 V2.0 Adapter für M.2 NGFF Disks - diesen Adapter will ich mal etwas austesten und hier darstellen. |
| |
| Vornweg - das native Booten von SSD ist einfach unglaublich schnell. Folgende Linux Version wird für den Test genutzt: |
| <code bash> |
| root@pi4test:~# lsb_release -a |
| No LSB modules are available. |
| Distributor ID: Raspbian |
| Description: Raspbian GNU/Linux 11 (bullseye) |
| Release: 11 |
| Codename: bullseye |
| </code> |
| |
| ==== Tests mit 128GB SSD Nativ ==== |
| |
| Ergebnisse Übersicht |
| ^ IO Art ^ Durchsatz ^ IOs ^ typ. Latenz ^ |
| | Seq. Write | 104 MB/s| 25300 IOs/s| 20µs| |
| | Seq. Read | 154 MB/s| 37600 IOs/s| 10µs| |
| | Seq. ReadWrite | 53,1 RMB/s| 12000 RIOs/s| 10-20µs| |
| | ::: | 53,1 WMB/s| 12000 WIOs/s| ::: | |
| | Rand. Write | 11,3 MB/s| 2769 IOs/s| 20 - 50µs| |
| | Rand. Read | 11,7 MB/s| 2848 IOs/s| 500µs| |
| | Rand. ReadWrite | 5552 RkB/s| 1355 RIOs/s| 20 - 500µs| |
| | ::: | 5546 WkB/s| 1353 WIOs/s| ::: | |
| |
| |
| === Sequential Test === |
| |
| Sequential Write: |
| <code bash> |
| root@pi4test:~# fio --rw=write --name=testfile --size 16G |
| testfile: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [W(1)][100.0%][w=91.7MiB/s][w=23.5k IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1124: Sat Apr 2 11:30:55 2022 |
| write: IOPS=25.3k, BW=98.9MiB/s (104MB/s)(16.0GiB/165685msec); 0 zone resets |
| clat (usec): min=8, max=717579, avg=35.32, stdev=1272.33 |
| lat (usec): min=8, max=717581, avg=36.07, stdev=1272.34 |
| clat percentiles (usec): |
| | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 11], |
| | 30.00th=[ 11], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 12], |
| | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 18], 95.00th=[ 20], |
| | 99.00th=[ 29], 99.50th=[ 38], 99.90th=[11994], 99.95th=[13960], |
| | 99.99th=[18482] |
| bw ( KiB/s): min= 4928, max=242056, per=100.00%, avg=101397.10, stdev=43117.26, samples=330 |
| iops : min= 1232, max=60514, avg=25349.28, stdev=10779.32, samples=330 |
| lat (usec) : 10=4.02%, 20=91.45%, 50=4.21%, 100=0.11%, 250=0.05% |
| lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.13%, 50=0.01% |
| lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% |
| cpu : usr=7.81%, sys=34.74%, ctx=11204, majf=0, minf=59 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| WRITE: bw=98.9MiB/s (104MB/s), 98.9MiB/s-98.9MiB/s (104MB/s-104MB/s), io=16.0GiB (17.2GB), run=165685-165685msec |
| |
| Disk stats (read/write): |
| sda: ios=1/32713, merge=0/37, ticks=0/8077384, in_queue=8077703, util=98.84% |
| </code> |
| |
| Sequential Read: |
| <code bash> |
| root@pi4test:~# fio --rw=read --name=testfile --size 16G |
| testfile: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [R(1)][100.0%][r=128MiB/s][r=32.8k IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1365: Sat Apr 2 12:06:42 2022 |
| read: IOPS=37.6k, BW=147MiB/s (154MB/s)(16.0GiB/111547msec) |
| clat (usec): min=6, max=492235, avg=21.61, stdev=2208.68 |
| lat (usec): min=6, max=492235, avg=22.21, stdev=2208.69 |
| clat percentiles (usec): |
| | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 9], |
| | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 9], |
| | 70.00th=[ 10], 80.00th=[ 10], 90.00th=[ 11], 95.00th=[ 12], |
| | 99.00th=[ 15], 99.50th=[ 16], 99.90th=[ 67], 99.95th=[ 260], |
| | 99.99th=[ 1106] |
| bw ( KiB/s): min=24248, max=262653, per=100.00%, avg=161982.49, stdev=72674.80, samples=207 |
| iops : min= 6062, max=65663, avg=40495.63, stdev=18168.66, samples=207 |
| lat (usec) : 10=89.05%, 20=10.66%, 50=0.15%, 100=0.07%, 250=0.02% |
| lat (usec) : 500=0.04%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% |
| lat (msec) : 250=0.01%, 500=0.01% |
| cpu : usr=8.23%, sys=56.34%, ctx=12287, majf=0, minf=30 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=4194304,0,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=16.0GiB (17.2GB), run=111547-111547msec |
| |
| Disk stats (read/write): |
| sda: ios=33442/3, merge=3/3, ticks=2496698/287, in_queue=2496987, util=47.90% |
| </code> |
| |
| Sequential ReadWrite: |
| <code bash> |
| root@pi4test:~# fio --rw=readwrite --name=testfile --size 16G |
| testfile: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [M(1)][100.0%][r=1617KiB/s,w=1865KiB/s][r=404,w=466 IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1648: Sat Apr 2 13:15:27 2022 |
| read: IOPS=12.0k, BW=50.7MiB/s (53.1MB/s)(8196MiB/161797msec) |
| clat (usec): min=6, max=1561.3k, avg=44.31, stdev=6185.70 |
| lat (usec): min=6, max=1561.3k, avg=44.92, stdev=6185.70 |
| clat percentiles (usec): |
| | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], |
| | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 9], |
| | 70.00th=[ 10], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], |
| | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 61], 99.95th=[ 172], |
| | 99.99th=[ 873] |
| bw ( KiB/s): min= 8, max=161920, per=100.00%, avg=72898.53, stdev=42363.84, samples=230 |
| iops : min= 2, max=40480, avg=18224.61, stdev=10591.00, samples=230 |
| write: IOPS=12.0k, BW=50.6MiB/s (53.1MB/s)(8188MiB/161797msec); 0 zone resets |
| clat (usec): min=6, max=267581, avg=22.01, stdev=944.13 |
| lat (usec): min=6, max=267582, avg=22.71, stdev=944.15 |
| clat percentiles (usec): |
| | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 9], |
| | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 9], |
| | 70.00th=[ 10], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], |
| | 99.00th=[ 21], 99.50th=[ 30], 99.90th=[ 178], 99.95th=[13304], |
| | 99.99th=[22152] |
| bw ( KiB/s): min= 56, max=162224, per=100.00%, avg=73784.31, stdev=41711.19, samples=227 |
| iops : min= 14, max=40556, avg=18446.04, stdev=10427.83, samples=227 |
| lat (usec) : 10=83.44%, 20=15.58%, 50=0.82%, 100=0.06%, 250=0.03% |
| lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.01% |
| lat (msec) : 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01% |
| cpu : usr=6.77%, sys=35.65%, ctx=8774, majf=0, minf=49 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=2098266,2096038,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=50.7MiB/s (53.1MB/s), 50.7MiB/s-50.7MiB/s (53.1MB/s-53.1MB/s), io=8196MiB (8594MB), run=161797-161797msec |
| WRITE: bw=50.6MiB/s (53.1MB/s), 50.6MiB/s-50.6MiB/s (53.1MB/s-53.1MB/s), io=8188MiB (8585MB), run=161797-161797msec |
| |
| Disk stats (read/write): |
| sda: ios=16929/16403, merge=2/38, ticks=3976787/2720784, in_queue=6697684, util=92.86% |
| </code> |
| |
| === Random Tests === |
| |
| Random Write: |
| <code bash> |
| root@pi4test:~# fio --rw=randwrite --name=testfile --size 16G |
| testfile: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [w(1)][100.0%][w=14.5MiB/s][w=3715 IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1405: Sat Apr 2 12:36:02 2022 |
| write: IOPS=2769, BW=10.8MiB/s (11.3MB/s)(16.0GiB/1514667msec); 0 zone resets |
| clat (usec): min=8, max=5130.9k, avg=352.32, stdev=28060.03 |
| lat (usec): min=9, max=5130.9k, avg=353.44, stdev=28060.04 |
| clat percentiles (usec): |
| | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], |
| | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 19], 60.00th=[ 21], |
| | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 30], |
| | 99.00th=[ 86], 99.50th=[17695], 99.90th=[18744], 99.95th=[19006], |
| | 99.99th=[27919] |
| bw ( KiB/s): min= 8, max=150741, per=100.00%, avg=21022.85, stdev=24157.80, samples=1596 |
| iops : min= 2, max=37685, avg=5255.66, stdev=6039.46, samples=1596 |
| lat (usec) : 10=0.75%, 20=53.96%, 50=43.53%, 100=0.80%, 250=0.12% |
| lat (usec) : 500=0.06%, 750=0.03%, 1000=0.02% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.70%, 50=0.02% |
| lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2000=0.01%, >=2000=0.01% |
| cpu : usr=1.56%, sys=5.94%, ctx=43410, majf=0, minf=27 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| WRITE: bw=10.8MiB/s (11.3MB/s), 10.8MiB/s-10.8MiB/s (11.3MB/s-11.3MB/s), io=16.0GiB (17.2GB), run=1514667-1514667msec |
| |
| Disk stats (read/write): |
| sda: ios=23/4041732, merge=0/385, ticks=5863/83180908, in_queue=83193169, util=99.41% |
| </code> |
| |
| Random Read |
| <code bash> |
| root@pi4test:~# fio --rw=randread --name=testfile --size 16G |
| testfile: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [r(1)][100.0%][r=11.4MiB/s][r=2916 IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1504: Sat Apr 2 13:09:22 2022 |
| read: IOPS=2848, BW=11.1MiB/s (11.7MB/s)(16.0GiB/1472703msec) |
| clat (usec): min=170, max=74926, avg=337.03, stdev=57.00 |
| lat (usec): min=171, max=74927, avg=338.26, stdev=57.03 |
| clat percentiles (usec): |
| | 1.00th=[ 260], 5.00th=[ 277], 10.00th=[ 289], 20.00th=[ 297], |
| | 30.00th=[ 306], 40.00th=[ 318], 50.00th=[ 326], 60.00th=[ 355], |
| | 70.00th=[ 367], 80.00th=[ 383], 90.00th=[ 396], 95.00th=[ 412], |
| | 99.00th=[ 429], 99.50th=[ 433], 99.90th=[ 449], 99.95th=[ 465], |
| | 99.99th=[ 498] |
| bw ( KiB/s): min= 6072, max=13192, per=100.00%, avg=11479.51, stdev=308.82, samples=2926 |
| iops : min= 1518, max= 3298, avg=2869.57, stdev=77.23, samples=2926 |
| lat (usec) : 250=0.05%, 500=99.94%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%, 100=0.01% |
| cpu : usr=3.30%, sys=24.04%, ctx=4200629, majf=0, minf=25 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=4194304,0,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=11.1MiB/s (11.7MB/s), 11.1MiB/s-11.1MiB/s (11.7MB/s-11.7MB/s), io=16.0GiB (17.2GB), run=1472703-1472703msec |
| |
| Disk stats (read/write): |
| sda: ios=4194136/52, merge=0/13, ticks=1223682/255, in_queue=1223958, util=99.45% |
| </code> |
| |
| Random ReadWrite: |
| <code bash> |
| root@pi4test:~# fio --rw=randrw --name=testfile --size 16G |
| testfile: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [m(1)][99.9%][r=4344KiB/s,w=4284KiB/s][r=1086,w=1071 IOPS][eta 00m:01s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1664: Sat Apr 2 13:43:09 2022 |
| read: IOPS=1355, BW=5422KiB/s (5552kB/s)(8196MiB/1548099msec) |
| clat (usec): min=221, max=669084, avg=662.63, stdev=12047.16 |
| lat (usec): min=222, max=669084, avg=663.74, stdev=12047.19 |
| clat percentiles (usec): |
| | 1.00th=[ 253], 5.00th=[ 273], 10.00th=[ 281], 20.00th=[ 293], |
| | 30.00th=[ 302], 40.00th=[ 314], 50.00th=[ 326], 60.00th=[ 351], |
| | 70.00th=[ 363], 80.00th=[ 375], 90.00th=[ 392], 95.00th=[ 412], |
| | 99.00th=[ 433], 99.50th=[ 461], 99.90th=[ 14615], 99.95th=[371196], |
| | 99.99th=[517997] |
| bw ( KiB/s): min= 7, max=12304, per=100.00%, avg=5621.54, stdev=4267.16, samples=2986 |
| iops : min= 1, max= 3076, avg=1405.32, stdev=1066.79, samples=2986 |
| write: IOPS=1353, BW=5416KiB/s (5546kB/s)(8188MiB/1548099msec); 0 zone resets |
| clat (usec): min=9, max=3560.6k, avg=52.69, stdev=7256.56 |
| lat (usec): min=9, max=3560.6k, avg=53.98, stdev=7256.59 |
| clat percentiles (usec): |
| | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 20], |
| | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 23], |
| | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 28], 95.00th=[ 33], |
| | 99.00th=[ 46], 99.50th=[ 51], 99.90th=[ 87], 99.95th=[ 120], |
| | 99.99th=[ 1811] |
| bw ( KiB/s): min= 7, max=13016, per=100.00%, avg=6402.52, stdev=3956.59, samples=2619 |
| iops : min= 1, max= 3254, avg=1600.56, stdev=989.15, samples=2619 |
| lat (usec) : 10=0.01%, 20=20.96%, 50=28.74%, 100=0.23%, 250=0.14% |
| lat (usec) : 500=49.74%, 750=0.09%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.01% |
| lat (msec) : 100=0.01%, 250=0.01%, 500=0.02%, 750=0.02%, 1000=0.01% |
| lat (msec) : 2000=0.01%, >=2000=0.01% |
| cpu : usr=2.53%, sys=14.07%, ctx=2114381, majf=0, minf=37 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=2098266,2096038,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=5422KiB/s (5552kB/s), 5422KiB/s-5422KiB/s (5552kB/s-5552kB/s), io=8196MiB (8594MB), run=1548099-1548099msec |
| WRITE: bw=5416KiB/s (5546kB/s), 5416KiB/s-5416KiB/s (5546kB/s-5546kB/s), io=8188MiB (8585MB), run=1548099-1548099msec |
| |
| Disk stats (read/write): |
| sda: ios=2097615/2048347, merge=0/326, ticks=1298235/40511328, in_queue=41812500, util=99.77% |
| </code> |
| |
| ==== Tests mit SD Card Samsung Evo Plus 32Gb==== |
| Die Evo Plus Serie soll besonders robust und performant sein. Deswegen die gleichen Tests als Vergleich zum SSD Modul. |
| |
| Die Random Tests habe ich mit verkleinerter Testdatei (nur 1Gb statt 16Gb) machen müssen. Der Random Write Test hat mehr als 11 Stunden Laufzeit bei 16GB hochgerechnet! Durch besseres Caching sind die Daten dadurch evtl. nicht richtig vergleichbar. Aber schon die seq. Tests zeigen deutliche Unterschiede. |
| |
| Ergebnisse Übersicht |
| ^ IO Art ^ Durchsatz ^ IOs ^ typ. Latenz ^ |
| | Seq. Write | 24,4 MB/s| 5948 IOs/s| 20 - 50µs| |
| | Seq. Read | 42 MB/s| 10800 IOs/s| 20µs| |
| | Seq. ReadWrite | 15,9 RMB/s| 3877 RIOs/s| 10-20µs| |
| | ::: | 15,9 WMB/s| 3873 WIOs/s| ::: | |
| | Rand. Write | 2246 kB/s| 548 IOs/s| 50µs| |
| | Rand. Read | 5149kB/s| 1257 IOs/s| 500 - 750µs| |
| | Rand. ReadWrite | 836 RkB/s| 204 RIOs/s| 50 - 750µs| |
| | ::: | 836 WkB/s| 204 WIOs/s| ::: | |
| |
| |
| === Sequential Test === |
| |
| Sequential Write: |
| <code bash> |
| root@pi4test:~# fio --rw=write --name=testfile --size 16G; |
| testfile: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| testfile: Laying out IO file (1 file / 16384MiB) |
| Jobs: 1 (f=1): [W(1)][100.0%][w=24.2MiB/s][w=6191 IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1512: Sat Apr 2 16:06:42 2022 |
| write: IOPS=5948, BW=23.2MiB/s (24.4MB/s)(16.0GiB/705139msec); 0 zone resets |
| clat (usec): min=9, max=4123.6k, avg=162.67, stdev=11836.43 |
| lat (usec): min=10, max=4123.6k, avg=163.63, stdev=11836.44 |
| clat percentiles (usec): |
| | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], |
| | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 22], |
| | 70.00th=[ 23], 80.00th=[ 26], 90.00th=[ 30], 95.00th=[ 34], |
| | 99.00th=[ 50], 99.50th=[ 104], 99.90th=[17433], 99.95th=[18744], |
| | 99.99th=[28181] |
| bw ( KiB/s): min= 8, max=142984, per=100.00%, avg=32273.09, stdev=23835.09, samples=1040 |
| iops : min= 2, max=35746, avg=8068.18, stdev=5958.77, samples=1040 |
| lat (usec) : 10=0.01%, 20=54.59%, 50=44.43%, 100=0.48%, 250=0.02% |
| lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.43%, 50=0.03% |
| lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01% |
| lat (msec) : >=2000=0.01% |
| cpu : usr=2.78%, sys=11.96%, ctx=32209, majf=0, minf=66 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| WRITE: bw=23.2MiB/s (24.4MB/s), 23.2MiB/s-23.2MiB/s (24.4MB/s-24.4MB/s), io=16.0GiB (17.2GB), run=705139-705139msec |
| |
| Disk stats (read/write): |
| mmcblk0: ios=81/33465, merge=151/1556, ticks=14287/48845811, in_queue=48860098, util=98.46% |
| </code> |
| |
| Sequential Read: |
| <code bash> |
| root@pi4test:~# fio --rw=read --name=testfile --size 16G; |
| testfile: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [R(1)][99.7%][r=43.0MiB/s][r=11.0k IOPS][eta 00m:01s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1548: Sat Apr 2 16:13:14 2022 |
| read: IOPS=10.8k, BW=42.0MiB/s (44.1MB/s)(16.0GiB/389870msec) |
| clat (usec): min=5, max=72870, avg=85.88, stdev=588.88 |
| lat (usec): min=6, max=72871, avg=86.83, stdev=588.88 |
| clat percentiles (usec): |
| | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 12], |
| | 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 13], |
| | 70.00th=[ 13], 80.00th=[ 13], 90.00th=[ 13], 95.00th=[ 14], |
| | 99.00th=[ 4686], 99.50th=[ 4752], 99.90th=[ 4948], 99.95th=[ 5145], |
| | 99.99th=[ 5276] |
| bw ( KiB/s): min=28640, max=44544, per=100.00%, avg=43925.71, stdev=627.86, samples=764 |
| iops : min= 7160, max=11136, avg=10981.31, stdev=157.00, samples=764 |
| lat (usec) : 10=5.15%, 20=93.15%, 50=0.11%, 100=0.03%, 250=0.01% |
| lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.01%, 4=0.01%, 10=1.56%, 20=0.01%, 100=0.01% |
| cpu : usr=4.41%, sys=21.28%, ctx=70088, majf=0, minf=30 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=4194304,0,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=42.0MiB/s (44.1MB/s), 42.0MiB/s-42.0MiB/s (44.1MB/s-44.1MB/s), io=16.0GiB (17.2GB), run=389870-389870msec |
| |
| Disk stats (read/write): |
| mmcblk0: ios=65512/71, merge=0/3, ticks=671153/40874, in_queue=712026, util=98.45% |
| </code> |
| |
| Sequential ReadWrite: |
| <code bash> |
| root@pi4test:~# fio --rw=readwrite --name=testfile --size 16G; |
| testfile: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [M(1)][99.6%][eta 00m:02s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=1560: Sat Apr 2 16:22:15 2022 |
| read: IOPS=3877, BW=15.1MiB/s (15.9MB/s)(8196MiB/541118msec) |
| clat (usec): min=5, max=3310.5k, avg=223.88, stdev=12260.46 |
| lat (usec): min=6, max=3310.5k, avg=224.58, stdev=12260.46 |
| clat percentiles (usec): |
| | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 8], |
| | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 8], |
| | 70.00th=[ 9], 80.00th=[ 13], 90.00th=[ 16], 95.00th=[ 19], |
| | 99.00th=[ 4047], 99.50th=[ 4293], 99.90th=[ 4555], 99.95th=[ 5014], |
| | 99.99th=[350225] |
| bw ( KiB/s): min= 7, max=105344, per=100.00%, avg=19911.38, stdev=13472.37, samples=841 |
| iops : min= 1, max=26336, avg=4977.78, stdev=3368.12, samples=841 |
| write: IOPS=3873, BW=15.1MiB/s (15.9MB/s)(8188MiB/541118msec); 0 zone resets |
| clat (usec): min=7, max=3855.8k, avg=22.92, stdev=4011.07 |
| lat (usec): min=8, max=3855.8k, avg=23.70, stdev=4011.07 |
| clat percentiles (usec): |
| | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 10], |
| | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 11], |
| | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 22], 95.00th=[ 25], |
| | 99.00th=[ 32], 99.50th=[ 38], 99.90th=[ 66], 99.95th=[ 108], |
| | 99.99th=[13304] |
| bw ( KiB/s): min= 7, max=107312, per=100.00%, avg=19889.49, stdev=13495.28, samples=841 |
| iops : min= 1, max=26828, avg=4972.32, stdev=3373.83, samples=841 |
| lat (usec) : 10=58.60%, 20=33.06%, 50=7.47%, 100=0.09%, 250=0.01% |
| lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2=0.02%, 4=0.19%, 10=0.50%, 20=0.01%, 50=0.01% |
| lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2000=0.01%, >=2000=0.01% |
| cpu : usr=2.81%, sys=11.54%, ctx=35531, majf=0, minf=46 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=2098266,2096038,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=8196MiB (8594MB), run=541118-541118msec |
| WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=8188MiB (8585MB), run=541118-541118msec |
| |
| Disk stats (read/write): |
| mmcblk0: ios=30739/16651, merge=687/127, ticks=638495/11316064, in_queue=11954560, util=98.92% |
| </code> |
| |
| === Random Tests === |
| |
| Random Write: |
| <code bash> |
| root@pi4test:~# fio --rw=randwrite --name=testfile --size 1G |
| testfile: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [w(1)][100.0%][w=3843KiB/s][w=960 IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=822: Sat Apr 2 18:20:04 2022 |
| write: IOPS=548, BW=2194KiB/s (2246kB/s)(1024MiB/477994msec); 0 zone resets |
| clat (usec): min=10, max=10665k, avg=1812.56, stdev=34779.32 |
| lat (usec): min=11, max=10665k, avg=1814.21, stdev=34779.33 |
| clat percentiles (usec): |
| | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 19], |
| | 20.00th=[ 27], 30.00th=[ 28], 40.00th=[ 30], |
| | 50.00th=[ 32], 60.00th=[ 35], 70.00th=[ 36], |
| | 80.00th=[ 36], 90.00th=[ 43], 95.00th=[ 52], |
| | 99.00th=[ 58459], 99.50th=[ 68682], 99.90th=[ 98042], |
| | 99.95th=[ 152044], 99.99th=[1199571] |
| bw ( KiB/s): min= 8, max=196686, per=100.00%, avg=2527.76, stdev=6814.11, samples=829 |
| iops : min= 2, max=49171, avg=631.78, stdev=1703.51, samples=829 |
| lat (usec) : 20=10.10%, 50=84.40%, 100=2.80%, 250=0.18%, 500=0.01% |
| lat (usec) : 750=0.01%, 1000=0.01% |
| lat (msec) : 4=0.01%, 20=0.01%, 50=1.35%, 100=1.07%, 250=0.06% |
| lat (msec) : 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%, >=2000=0.01% |
| cpu : usr=0.56%, sys=1.79%, ctx=8529, majf=1, minf=27 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| WRITE: bw=2194KiB/s (2246kB/s), 2194KiB/s-2194KiB/s (2246kB/s-2246kB/s), io=1024MiB (1074MB), run=477994-477994msec |
| |
| Disk stats (read/write): |
| mmcblk0: ios=4/177602, merge=0/231, ticks=240/59463236, in_queue=59463477, util=95.64% |
| </code> |
| |
| Random Read |
| <code bash> |
| root@pi4test:~# fio --rw=randread --name=testfile --size 1G |
| testfile: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 |
| fio-3.25 |
| Starting 1 process |
| Jobs: 1 (f=1): [r(1)][100.0%][r=8740KiB/s][r=2185 IOPS][eta 00m:00s] |
| testfile: (groupid=0, jobs=1): err= 0: pid=845: Sat Apr 2 18:23:36 2022 |
| read: IOPS=1257, BW=5028KiB/s (5149kB/s)(1024MiB/208547msec) |
| clat (usec): min=364, max=1166.1k, avg=565.39, stdev=2278.64 |
| lat (usec): min=365, max=1166.1k, avg=566.64, stdev=2278.64 |
| clat percentiles (usec): |
| | 1.00th=[ 392], 5.00th=[ 408], 10.00th=[ 412], 20.00th=[ 457], |
| | 30.00th=[ 570], 40.00th=[ 578], 50.00th=[ 594], 60.00th=[ 594], |
| | 70.00th=[ 611], 80.00th=[ 635], 90.00th=[ 644], 95.00th=[ 652], |
| | 99.00th=[ 668], 99.50th=[ 676], 99.90th=[ 758], 99.95th=[ 840], |
| | 99.99th=[ 3851] |
| bw ( KiB/s): min= 360, max= 8776, per=100.00%, avg=6948.06, stdev=521.44, samples=301 |
| iops : min= 90, max= 2194, avg=1736.77, stdev=130.29, samples=301 |
| lat (usec) : 500=28.14%, 750=71.76%, 1000=0.07% |
| lat (msec) : 2=0.01%, 4=0.03%, 10=0.01%, 2000=0.01% |
| cpu : usr=1.51%, sys=7.27%, ctx=264364, majf=0, minf=24 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=5028KiB/s (5149kB/s), 5028KiB/s-5028KiB/s (5149kB/s-5149kB/s), io=1024MiB (1074MB), run=208547-208547msec |
| |
| Disk stats (read/write): |
| mmcblk0: ios=261731/11075, merge=0/81, ticks=136444/6987958, in_queue=7124403, util=99.22% |
| </code> |
| |
| Random ReadWrite: |
| <code bash> |
| root@pi4test:~# fio --rw=randrw --name=testfile --size 1G |
| testfile: (groupid=0, jobs=1): err= 0: pid=853: Sat Apr 2 18:34:18 2022 |
| read: IOPS=204, BW=816KiB/s (836kB/s)(512MiB/642113msec) |
| clat (usec): min=318, max=15370k, avg=4536.72, stdev=125745.82 |
| lat (usec): min=319, max=15370k, avg=4537.91, stdev=125745.85 |
| clat percentiles (usec): |
| | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 424], |
| | 20.00th=[ 441], 30.00th=[ 529], 40.00th=[ 586], |
| | 50.00th=[ 594], 60.00th=[ 603], 70.00th=[ 611], |
| | 80.00th=[ 619], 90.00th=[ 627], 95.00th=[ 635], |
| | 99.00th=[ 701], 99.50th=[ 94897], 99.90th=[1266680], |
| | 99.95th=[2432697], 99.99th=[4731175] |
| bw ( KiB/s): min= 7, max= 8232, per=100.00%, avg=2077.54, stdev=2986.64, samples=505 |
| iops : min= 1, max= 2058, avg=519.32, stdev=746.63, samples=505 |
| write: IOPS=204, BW=817KiB/s (836kB/s)(512MiB/642113msec); 0 zone resets |
| clat (usec): min=11, max=14314k, avg=338.08, stdev=57482.73 |
| lat (usec): min=12, max=14314k, avg=339.48, stdev=57482.74 |
| clat percentiles (usec): |
| | 1.00th=[ 15], 5.00th=[ 22], 10.00th=[ 23], 20.00th=[ 24], |
| | 30.00th=[ 24], 40.00th=[ 25], 50.00th=[ 27], 60.00th=[ 27], |
| | 70.00th=[ 28], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 38], |
| | 99.00th=[ 48], 99.50th=[ 57], 99.90th=[ 90], 99.95th=[ 104], |
| | 99.99th=[109577] |
| bw ( KiB/s): min= 7, max= 8520, per=100.00%, avg=2517.12, stdev=3121.97, samples=417 |
| iops : min= 1, max= 2130, avg=629.21, stdev=780.47, samples=417 |
| lat (usec) : 20=0.73%, 50=48.93%, 100=0.33%, 250=0.01%, 500=14.52% |
| lat (usec) : 750=35.06%, 1000=0.04% |
| lat (msec) : 2=0.01%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% |
| lat (msec) : 100=0.15%, 250=0.12%, 500=0.01%, 750=0.01%, 1000=0.01% |
| lat (msec) : 2000=0.03%, >=2000=0.04% |
| cpu : usr=0.38%, sys=1.74%, ctx=131377, majf=0, minf=39 |
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% |
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% |
| issued rwts: total=131040,131104,0,0 short=0,0,0,0 dropped=0,0,0,0 |
| latency : target=0, window=0, percentile=100.00%, depth=1 |
| |
| Run status group 0 (all jobs): |
| READ: bw=816KiB/s (836kB/s), 816KiB/s-816KiB/s (836kB/s-836kB/s), io=512MiB (537MB), run=642113-642113msec |
| WRITE: bw=817KiB/s (836kB/s), 817KiB/s-817KiB/s (836kB/s-836kB/s), io=512MiB (537MB), run=642113-642113msec |
| |
| Disk stats (read/write): |
| mmcblk0: ios=131067/107292, merge=30/113, ticks=593905/68646500, in_queue=69240405, util=96.72% |
| </code> |
| |
| |
| ===== CPU Performance Raspi4 Native vs VM ===== |
| |
| ==== Nativ ==== |
| |
| <code bash> |
| root@pi4test:~# 7z b |
| |
| 7-Zip [32] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21 |
| p7zip Version 16.02 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,32 bits,4 CPUs LE) |
| |
| LE |
| CPU Freq: 64000000 64000000 64000000 - - - - - 2048000000 |
| |
| RAM size: 7898 MB, # CPU hardware threads: 4 |
| RAM usage: 882 MB, # Benchmark threads: 4 |
| |
| Compressing | Decompressing |
| Dict Speed Usage R/U Rating | Speed Usage R/U Rating |
| KiB/s % MIPS MIPS | KiB/s % MIPS MIPS |
| |
| 22: 4113 361 1107 4001 | 108393 399 2318 9248 |
| 23: 3935 375 1070 4010 | 105545 398 2294 9132 |
| 24: 3777 376 1080 4062 | 102803 398 2265 9025 |
| 25: 3555 371 1095 4060 | 99861 398 2231 8887 |
| ---------------------------------- | ------------------------------ |
| Avr: 371 1088 4033 | 398 2277 9073 |
| Tot: 385 1682 6553 |
| root@pi4test:~# |
| |
| root@pi4test:~# 7z b |
| |
| 7-Zip [32] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21 |
| p7zip Version 16.02 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,32 bits,4 CPUs LE) |
| |
| LE |
| CPU Freq: 32000000 - - - 128000000 - - 1024000000 2048000000 |
| |
| RAM size: 7898 MB, # CPU hardware threads: 4 |
| RAM usage: 882 MB, # Benchmark threads: 4 |
| |
| Compressing | Decompressing |
| Dict Speed Usage R/U Rating | Speed Usage R/U Rating |
| KiB/s % MIPS MIPS | KiB/s % MIPS MIPS |
| |
| 22: 4043 350 1123 3933 | 108442 399 2317 9252 |
| 23: 3917 370 1078 3991 | 105547 398 2294 9132 |
| 24: 3778 374 1085 4062 | 102708 398 2267 9016 |
| 25: 3573 371 1101 4080 | 99778 398 2231 8880 |
| ---------------------------------- | ------------------------------ |
| Avr: 366 1097 4017 | 398 2277 9070 |
| Tot: 382 1687 6543 |
| |
| </code> |
| |
| |
===== Fortschrittsanzeige bei dd (automatisch ;-) ===== | ===== Fortschrittsanzeige bei dd (automatisch ;-) ===== |