Skip to content

Changes

Page history
(main) Exploration of IOR and FIO benchmarks; Noteful wiki authored by Parker Addison's avatar Parker Addison
Show whitespace changes
Inline Side-by-side
Notes.md
View page @ 9358c65a
Placeholder # Filesystem Benchmarks
\ No newline at end of file
###### 03/09-03/13 2021
## IOR
### Setup
First deploy the volume and a pod.
```
k create -f volumes/block.yml
k create -f minimal-deploy.yml
```
These commands are run in the pod.
Dependencies. See: https://github.com/hpc/ior/blob/main/testing/docker/ubuntu16.04/Dockerfile
```
apt-get update
apt-get install -y libopenmpi-dev openmpi-bin mpich git pkg-config gcc git vim less curl wget
apt-get install -y sudo
```
Downloading
```
wget -qO- https://github.com/hpc/ior/releases/download/3.3.0/ior-3.3.0.tar.gz | tar -zxv
```
Configuration. See `./configure --help`.
```
./configure
```
Installation
```
make
```
### Running
See: https://ior.readthedocs.io/en/latest/userDoc/tutorial.html
```
cd src
./ior ...
```
or
```
mpirun ...
```
Not sure how to really use it yet.
When I run ior it does a test instantly it seems.
When I tried doing
```
mpirun -n 64 ./ior -t 1m -b 16m -s 16
```
I got a ton of:
```
ior ERROR: open64("testFile", 66, 0664) failed, errno 13, Permission denied (aiori-POSIX.c:412)
...
[filebench-574869c787-pdn62:07749] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
...
```
Also note that I ran `useradd testu` and `su testu` because MPIrun doesn't want to be run as a root user. But this user has no permissions! I think that's the issue.
Seems like a `chmod -R 777 .` as the root fixed this!
For example, run 10 tasks with a transfer size of 1m(egabyte?), a block size of 16m(egabyte?), and a segment count of 16:
```
mpirun -n 10 ./src/ior -t 1m -b 16m -s 16
```
<details>
<summary>
Output:
</summary>
```raw
IOR-3.3.0: MPI Coordinated Test of Parallel I/O
Began : Mon Mar 8 23:08:17 2021
Command line : ./src/ior -t 1m -b 16m -s 16
Machine : Linux filebench-574869c787-pdn62
TestID : 0
StartTime : Mon Mar 8 23:08:17 2021
Path : /storage/ior-3.3.0
FS : 8.0 GiB Used FS: 0.5% Inodes: 4.0 Mi Used Inodes: 0.0%
Options:
api : POSIX
apiVersion :
test filename : testFile
access : single-shared-file
type : independent
segments : 16
ordering in a file : sequential
ordering inter file : no tasks offsets
nodes : 1
tasks : 10
clients per node : 10
repetitions : 1
xfersize : 1 MiB
blocksize : 16 MiB
aggregate filesize : 2.50 GiB
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
write 678.06 678.07 0.118931 16384 1024.00 0.770489 3.78 3.59 3.78 0
read 4233 4234 0.019146 16384 1024.00 0.000035 0.604695 0.298354 0.604706 0
remove - - - - - - - - 3.20 0
Max Write: 678.06 MiB/sec (711.00 MB/sec)
Max Read: 4233.46 MiB/sec (4439.11 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 678.06 678.06 678.06 0.00 678.06 678.06 678.06 0.00 3.77548 NA NA 0 10 10 1 0 0 1 0 0 16 16777216 1048576 2560.0 POSIX 0
read 4233.46 4233.46 4233.46 0.00 4233.46 4233.46 4233.46 0.00 0.60471 NA NA 0 10 10 1 0 0 1 0 0 16 16777216 1048576 2560.0 POSIX 0
Finished : Mon Mar 8 23:08:25 2021
```
</details>
## IO-500
### Setup
The hpc/ior:ubuntu16.04 image (built locally and pushed to Docker Hub parkeraddison/ior:ubuntu16.04) *almost* passes `./prepare.sh` for the io500 repo -- it just needs to
```
apt-get install -y autoconf
```
Set up permissions
```
groupadd stor
chgrp -R stor /storage
chmod -R g+rwx /storage
```
```
useradd usr -G stor
```
### Running
```
su usr
mpiexec -np 2 ./io500 config-minimal.ini
```
<details>
<summary>Output:</summary>
```
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LJJ3QEF6WLMS4VWPVV2XKL6JYS:/var/lib/docker/overlay2/l/24YRTXTAGXIULRRXZY4JB5WTGG:/var/lib/docker/overlay2/l/42MQ3LQTYO2IUBUARVOT7IPDQP:/var/lib/docker/overlay2/l/YGOPACSWOGMEMKHTJYCM6UU3FH:/var/lib/docker/overlay2/l/KPZVRXHJW6K2V5FL24TRWFWO6B:/var/lib/docker/overlay2/l/HX22FHOBPYU4GIEFU6V5JWP2FJ:/var/lib/docker/overlay2/l/GJP2A7A4T3XQYPZHHNZR3LC76R:/var/lib/docker/overlay2/l/TZLAGOYFXJHZETSVMY4KIDZ543:/var/lib/docker/overlay2/l/4NTW7PG2N53XK'
IO500 version io500-sc20_v3-6-gd25ea80d54c7
ERROR: write(12, 0x225c000, 2097152) failed, (aiori-POSIX.c:563)
```
</details>
Oof.
Look like: https://stackoverflow.com/questions/46138549/docker-openmpi-and-unexpected-end-of-proc-mounts-line
I'm trying a flattened image now (and including autoconf).
https://tuhrig.de/flatten-a-docker-container-or-image/
Still getting
```
./ior: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory
```
Seems like a dependency problem -- openmpi3 is needed. Tried the centos7 image and the same thing happened. Note that to use mpirun in centos you need to first run `module load mpi`. This image seems to work (locally). For some reason it caused an error when I tried to deploy on PRP. May try again.
Repo: https://github.com/joshuarobinson/docker_ior_mpi
###### 03/14
## Custom image for IO500 dependencies
I created a custom image to hold the ior/io500 dependencies so I'll have finer control over it. Then I went ahead and edited the `./ior --list > config-all.ini` output to disable all but the two easy IOR tests. I've put this into `config.ini`.
I also changed the transfer and block size to very small values (proof of concept). I believe in the past when I was trying to run it I was using the defaults (very large values!).
<details>
<summary>
```
bash-4.3$ mpiexec -np 2 ./io500 config.ini
```
</summary>
```
ERROR INVALID (src/phase_dbg.c)stonewall-time != 300s
IO500 version io500-sc20_v3-6-gd25ea80d54c7
[RESULT-invalid] ior-easy-write 0.650314 GiB/s : time 0.008 seconds
[RESULT-invalid] mdtest-easy-write 0.000000 kIOPS : time 0.000 seconds
[RESULT-invalid] ior-hard-write 0.000000 GiB/s : time 0.000 seconds
[RESULT-invalid] mdtest-hard-write 0.000000 kIOPS : time 0.000 seconds
[RESULT-invalid] find 0.000000 kIOPS : time 0.000 seconds
[RESULT] ior-easy-read 3.403407 GiB/s : time 0.003 seconds
[RESULT-invalid] mdtest-easy-stat 0.000000 kIOPS : time 0.000 seconds
[RESULT-invalid] ior-hard-read 0.000000 GiB/s : time 0.000 seconds
[RESULT-invalid] mdtest-hard-stat 0.000000 kIOPS : time 0.000 seconds
[RESULT-invalid] mdtest-easy-delete 0.000000 kIOPS : time 0.000 seconds
[RESULT-invalid] mdtest-hard-read 0.000000 kIOPS : time 0.000 seconds
[RESULT-invalid] mdtest-hard-delete 0.000000 kIOPS : time 0.000 seconds
[SCORE-invalid] Bandwidth 0.000000 GiB/s : IOPS 0.000000 kiops : TOTAL 0.000000
The result files are stored in the directory: ./results/2021.03.14-23.26.49
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 150 RUNNING AT filebench-78c6c98d98-nrdlr
= EXIT CODE: 139
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
```
</details>
Despite the warnings, I'm pretty sure it actually worked.
---
## FIO benchmark
Was able to get [FIO](https://fio.readthedocs.io/en/latest/fio_doc.html) running by downloading the tar.gz from https://github.com/axboe/fio, installing the dependencies [here](https://github.com/dmonakhov/docker-image--alpine-fio) (alpine), then running `make` (ignoring a warning) and `make install`.
Finally, I created a simple [job file](https://fio.readthedocs.io/en/latest/fio_doc.html#job-file-format) as `write.fio` then ran `fio write.fio`.
<details>
<summary>Output:</summary>
```
bash-4.3$ fio write.fio
job1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.26
Starting 1 process
job1: Laying out IO file (1 file / 128MiB)
job1: (groupid=0, jobs=1): err= 0: pid=4189: Sun Mar 14 23:51:53 2021
write: IOPS=273k, BW=1067MiB/s (1118MB/s)(128MiB/120msec); 0 zone resets
clat (nsec): min=1130, max=190684, avg=3212.52, stdev=8403.44
lat (nsec): min=1200, max=190734, avg=3271.35, stdev=8403.92
clat percentiles (nsec):
| 1.00th=[ 1304], 5.00th=[ 1352], 10.00th=[ 1384], 20.00th=[ 1464],
| 30.00th=[ 1544], 40.00th=[ 1624], 50.00th=[ 1688], 60.00th=[ 1768],
| 70.00th=[ 1896], 80.00th=[ 2096], 90.00th=[ 2512], 95.00th=[ 3280],
| 99.00th=[55552], 99.50th=[58624], 99.90th=[77312], 99.95th=[84480],
| 99.99th=[91648]
lat (usec) : 2=76.55%, 4=19.21%, 10=0.49%, 20=1.31%, 50=0.86%
lat (usec) : 100=1.58%, 250=0.01%
cpu : usr=10.08%, sys=89.92%, ctx=57, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1067MiB/s (1118MB/s), 1067MiB/s-1067MiB/s (1118MB/s-1118MB/s), io=128MiB (134MB), run=120-120msec
Disk stats (read/write):
rbd2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
```
</details>
### References, graphs, and job files
It looks like FIO is pretty popular! There are lots of repositories that have tools to work with FIO (both inputs and outputs).
Here are some repos which chart fio outputs or do other helpful things with FIO. They may all help get a better idea of how to understand the outputs and how to set up useful jobs!
- https://github.com/khailey/fio_scripts ([example output](https://sites.google.com/site/oraclemonitor/i-o-graphics#TOC-Percentile-Latency))
- https://github.com/louwrentius/fio-plot
- https://github.com/wallnerryan/fio-tools
- https://github.com/intel/fiovisualizer
- https://github.com/xridge/fio-docker
- https://github.com/pcuzner/fio-tools
- https://github.com/javigon/fio_tests
- https://github.com/meganerd/fio-examples
- https://github.com/jan--f/fio_graphs
- https://github.com/amefs/fio-bench
- https://github.com/mcgrof/fio-tests
- https://github.com/mchad1/fio-parser
- https://github.com/storpool/fio-tests
- https://github.com/perftool-incubator/bench-fio
This is probably a good search: https://github.com/search?q=fio+benchmark&type=Repositories
> ###### [957b70c](ssh///https://gitlab-ssh.nautilus.optiputer.net/30622/parkeraddison/filesystem-benchmarks/commit/957b70caaca3f180ca323dbf1045965823f349df)
> "Exploration of IOR and FIO benchmarks; Noteful wiki" | HEAD -> main | 2021-03-14