Notes.md

@@ -26,6 +26,9 @@
 * [Automated parameter sweeps](#automated-parameter-sweeps)
 * [Parsing and graphing IOR outputs](#parsing-and-graphing-ior-outputs)
 * [Some useful parameter sweeps](#some-useful-parameter-sweeps)
+  * [Side note: Darshan](#side-note-darshan)
+* [FIO on NAS](#fio-on-nas)
+* [Darshan on NAS](#darshan-on-nas)
 
 <!-- vim-markdown-toc -->
 
@@ -833,7 +836,69 @@ The current steps remaining are:
 
 ###### 04/07
 
-
 > ###### [246de9f](ssh///https://gitlab-ssh.nautilus.optiputer.net/30622/parkeraddison/filesystem-benchmarks/commit/246de9feb3882fe6a91a66306661a3aeeacf3814)
 > "More robust parameter sweep code and pbs job script; Add sweep and visualization to readme" | HEAD -> main | 2021-04-07
 
+###### 04/08
+
+For parameter sweeps, we have the following guidelines:
+- We should do **multiple iterations** of each test for consistency sake. We can change this depending on how long tests take, but for now maybe `-i 5` or so
+- We're pretty interested in how each system **scales with concurrency** -- so testing sweeping **# tasks** is an important test
+- We're interested in how ^ might change with **different data size** and **different access patterns** -- so we should test various **file size** (combination of block and segment size) and **transfer size**
+- Key importance: *to explore, to understand **what bottlenecks we might be experiencing***
+
+Largely, this is exploratory -- the parameter values and sweeps aren't fixed by any means, rather we should try some out and if we see something interesting then we should dive into it further. Fortunately, the tests (at the current scale I've tested) don't take too long (although now that multiple iterations are being run, expect it to take $i$ times longer).
+
+To come up with the initial values, though, I've been drawing inspiration mostly from some papers which have used IOR to evaluate HPC performance:
+- [Using IOR to Analyze the I/O performance for HPC Platforms](https://crd.lbl.gov/assets/pubs_presos/CDS/ATG/cug07shan.pdf) by Hongzhang Shan, John Shalf
+  - Sides: https://cug.org/5-publications/proceedings_attendee_lists/2007CD/S07_Proceedings/pages/Authors/Shan/Shan_slides.pdf
+  - Conducted **user survey for typical IO access patterns** at NERSC. Findings:
+    - Mostly sequential IO (rather than random)
+    - Mostly writes -- really? I would assume that most scientific projects are moreso write-once read-many...
+    - Transfer size varies a lot -- **"1KB to tens of MB"**
+    - Typical IO patterns: **one processor**, **one file per processor**, **MPI-IO single shared file**
+    - 
+- [I/O Performance on Cray XC30](https://www.nersc.gov/assets/pubs_presos/Edison-IO-CUG2014-paper-Zhao-4.pdf) by Zhengji Zhao, Doug Petesch, David Knaak, and Tina Declerck
+  - Sides: https://www.nersc.gov/assets/pubs_presos/Edison-IO-CUG2014-presentation.pdf
+  - 
+
+### Side note: Darshan
+
+Darshan is an IO profiler which intercepts IO calls to collect statistics which can be viewed on a timeline or summarized later -- things like bandwidth, IO size, etc. Basically, it's a way to get all of those useful measurements which a finished IOR/FIO run tells us *but on any arbitrary mpirun jobs (including scientific application benchmarks)!*
+
+Useful video: https://www.youtube.com/watch?v=7cDoBusXK5Q; slides: https://pop-coe.eu/sites/default/files/pop_files/darshan_io_profiling_webinar.pdf
+
+Definitely worth getting this to run on NAS -- even for IOR runs. The video mentions looking at how well the percentage of *metadata* IO scales, because that was a bottleneck they faced.
+
+###### 04/09
+
+## FIO on NAS
+
+It came to my attention that I've found a lot of academic papers which reference IOR, but not a lot of widespread 'internet' popularity. FIO, however, is immensely popular in terms of internet points -- plenty of blog posts, technical pages (from Microsoft, Google, Oracle, etc)... I wonder if there are some HPC papers which reference FIO?
+
+In order to run FIO on NAS, it the release can be downloaded and unpacked like so:
+```bash
+wget -O- https://github.com/axboe/fio/archive/refs/tags/fio-3.26.tar.gz | tar zxf -
+```
+Before we `make` however, we need to upgrade to gcc version of at least 4.9
+```bash
+module avail gcc
+...
+module load gcc/8.2
+cd fio-fio-3.26/
+make
+```
+The minimal job in [`readwrite.fio`](/fio/readwrite.fio) can be run with
+```bash
+path/to/fio path/to/readwrite.fio
+```
+
+Hmmmm, I came across [](https://dl.acm.org/doi/pdf/10.1145/2335755.2335789) by Shawn Strande, Pietro Cicotti, et al. (and it's out of SDSC - it's a small world after all ;) ) but, I also think I came across the reason why I'm not seeing HPC papers that use FIO: I'm not so sure that FIO can do single-shared-file workloads, https://github.com/axboe/fio/issues/631. So it might be really easy to set up job script and get baseline readings for your filesystems, but not when there are multiple nodes involved.
+
+## Darshan on NAS
+
+- [ ] TODO
+
+> ###### [b2beb35](ssh///https://gitlab-ssh.nautilus.optiputer.net/30622/parkeraddison/filesystem-benchmarks/commit/b2beb35c3771b5251442b3cdd48801f1d2c57e4c)
+> "FIO works on NAS" | HEAD -> main | 2021-04-09
+