A lot of storage is rated by the I/O Operations it can handle. This is a bit counter-intuitive when MB/s is a lot easier to measure. Where the difference comes is in 1000 2kb accesses vs 1000 32kb accesses. They both pound the storage quite a bit, but the later has a higher transfer rate. Storage bottlenecks more on random accesses than it does on raw throughput, at least with rotational media.
Figuring out the theoretical I/O operations a specific drive can handle is fairly simple. Lets take a look at a top tier 15K RPM 6GB SAS drive, the Seagate Cheetah 15K.7. The important stats are:
Note, capacity doesn't matter. The maximum theoretical I/O operations this drive can sustain is defined by the formula:
1000 / (average-latency+random-seek)
So for Write I/O, it can sustain 169 I/O ops, and for Read I/O it can sustain 185 I/O ops. When used in RAID configs, I/O operation capacity aggregates. Parity and mirror RAID will consume I/O ops as overhead though.
Significantly sequential I/O access patterns won't require the average-seek measurement, since the drive is just accessing the next block on the disk. In that case, the drive can pull off 500 I/O ops. As you can see, how many I/O ops a drive can actually sustain depends on the I/O access patterns.
Time for a real-world example! Our EVA6100 has 80 Fibre Channel drives in it that spin at 10K RPM. After I do the above math for one drive and then multiply by 80, I get a theoretical maximum I/O Operation capacity of 11,120. After doing some monitoring of IO Ops throughout a day, I find that we can do sustained IO ops of about 8K (those Exchange Online Defrags are very hard on storage!) and occasionally burst up to about 10.5K for very short periods (5-10 seconds).
Now for our EVA4400. It has 48 7.2K RPM drives in it. Its theoretical maximum I/O op count is about 3.5K. Doing the same monitoring as I did with the 6100 I see that we have one 15 minute period where we drove 4.3K I/O operations. Above max. How is this possible? At that time the SQL backups were firing which is a significantly sequential operation since that volume is not significantly fragmented. The theoretical max for 100% sequential operations on these drives is about 11.5K. The EVA architecture introduces a significant amount of randomization even in otherwise sequential access patterns, but enough pure sequential I/O happened to gain some of the benefits.
This is why finding a Solid State Drive capable of sustaining 25,000 I/O ops all by itself (Seagate again, just an example not an endorsement) is such a major thing. ONE DRIVE can out-perform our entire EVA6100. Of course, you only get 200GB on that drive where with our 6100 we have two orders of magnitude more storage.
As a theoretical exercise, if we filled the 4400 with those 200GB Seagate SSD drives, we'd get a maximum storage capacity of about 6.5TB, but it would theoretically be able to handle 1.2 million I/O operations. We'd hit controller limits well before we got that far. As it is right now, our controller CPU rarely moves above 25% even during busy times.
Figuring out the theoretical I/O operations a specific drive can handle is fairly simple. Lets take a look at a top tier 15K RPM 6GB SAS drive, the Seagate Cheetah 15K.7. The important stats are:
| Average Latency: | 2.0ms |
| Random seek read time: | 3.4ms |
| Random seek write time: | 3.9ms |
Note, capacity doesn't matter. The maximum theoretical I/O operations this drive can sustain is defined by the formula:
1000 / (average-latency+random-seek)
So for Write I/O, it can sustain 169 I/O ops, and for Read I/O it can sustain 185 I/O ops. When used in RAID configs, I/O operation capacity aggregates. Parity and mirror RAID will consume I/O ops as overhead though.
Significantly sequential I/O access patterns won't require the average-seek measurement, since the drive is just accessing the next block on the disk. In that case, the drive can pull off 500 I/O ops. As you can see, how many I/O ops a drive can actually sustain depends on the I/O access patterns.
Time for a real-world example! Our EVA6100 has 80 Fibre Channel drives in it that spin at 10K RPM. After I do the above math for one drive and then multiply by 80, I get a theoretical maximum I/O Operation capacity of 11,120. After doing some monitoring of IO Ops throughout a day, I find that we can do sustained IO ops of about 8K (those Exchange Online Defrags are very hard on storage!) and occasionally burst up to about 10.5K for very short periods (5-10 seconds).
Now for our EVA4400. It has 48 7.2K RPM drives in it. Its theoretical maximum I/O op count is about 3.5K. Doing the same monitoring as I did with the 6100 I see that we have one 15 minute period where we drove 4.3K I/O operations. Above max. How is this possible? At that time the SQL backups were firing which is a significantly sequential operation since that volume is not significantly fragmented. The theoretical max for 100% sequential operations on these drives is about 11.5K. The EVA architecture introduces a significant amount of randomization even in otherwise sequential access patterns, but enough pure sequential I/O happened to gain some of the benefits.
This is why finding a Solid State Drive capable of sustaining 25,000 I/O ops all by itself (Seagate again, just an example not an endorsement) is such a major thing. ONE DRIVE can out-perform our entire EVA6100. Of course, you only get 200GB on that drive where with our 6100 we have two orders of magnitude more storage.
As a theoretical exercise, if we filled the 4400 with those 200GB Seagate SSD drives, we'd get a maximum storage capacity of about 6.5TB, but it would theoretically be able to handle 1.2 million I/O operations. We'd hit controller limits well before we got that far. As it is right now, our controller CPU rarely moves above 25% even during busy times.
"So for Write I/O, it can sustain 185 I/O ops, and for Read I/O it can sustain 169 I/O ops." Are those numbers the wrong way around?
Indeed they are! This is what I get for not having an editor. Switched.