[haiku-development] Re: I/O Scheduler experiment
- From: waddlesplash <waddlesplash@xxxxxxxxx>
- To: Haiku Development ML <haiku-development@xxxxxxxxxxxxx>
- Date: Wed, 25 Sep 2019 20:00:09 -0400
On Wed, Sep 25, 2019 at 3:47 AM Kyle Ambroff-Kao <kyle@xxxxxxxxxxxxxx> wrote:
Re-sending this because I completely failed to follow the list
etiquette[1] document. Sorry about that. I've been lurking for a long
time and haven't sent mail to this list before so I totally forgot
about it. I suspect my initial mail got flagged.
Ah, it probably did; sorry, I haven't been keeping as close an eye on
the "error watches" emails these past few days (there were a number of
them due to some bounces.)
* All of the work IOSchedulerSimple is doing is probably pretty meaningless
on a
flash device that doesn't have a spinning platter.
This is not true. The goal is also to avoid one thread saturating I/O,
e.g. submitting a 100MB read or write request all at once. Nearly all
disk interfaces are sequential, so we need to break up such large
transfers so one process does not lock everything up.
In practice, I guess most of the consumers of the raw disk IO are the
filesystems, the more sophisticated of which will be using the cache
interfaces that will use page writebacks from a different thread than
the one actually doing reading/writing on the FS. But we still need to
be careful down here.
* Many call sites of IOScheduler::ScheduleRequest(IORequest*) just block on
the
request anyway, so the extra latency incurred by queueing and
context switching
between threads isn't worth it.
That latency is pretty low... are you sure about this?
* The single thread in IOSchedulerSimple may not be able to saturate most
modern
block devices even without the self throttling.
This could do with some benchmarking to determine exactly what the
saturation is, then :) I'd imagine it is different for different
drives (even across one interface.)
* It seems that for some block device drivers, such as nvme_disk,
IOSchedulerSimple is not used. Which seems appropriate.
AFAIK, nvme_disk is the only one like this (and the fact that it isn't
was a design decision on my part), because unlike virtually all other
disk interfaces, NVMe supports multiple I/O queues so we can just
allocate a bunch of them and round-robin requests to it, which in
practice is much higher performance than trying to schedule stuff most
of the time.
Unfortunately it seems that at least some parts of the system assume
IO will be executed sequentially, not in parallel (or some assumption
related to this one), because BFS and the block cache under NVMe has
been reported to KDL when untar'ing large archives
(
https://dev.haiku-os.org/ticket/15123 -- though this was on VMware, I
actually haven't heard any reports from bare metal about this but
there are very few people running the NVMe driver on bare metal I
think.)
It's unclear to me whether adding more to the IOScheduler interface
breaks compatibility in any way. It seems to only be an internal API
and therefore safe to touch change arbitrarily, but you all are the
experts here.
Yes, IOScheduler is a private kernel API, so we can change it as we
like (for now.)
That being said, the
nvme_disk driver seems to do the right thing here, so I think it's
probably a good idea to use a similar strategy for SATA SSDs as well.
Can we really queue commands to SATA SSDs concurrently? (I admittedly
know very little about SATA; and I'd be surprised if our driver for it
implemented that if it did.) If so, then perhaps this would be a good
idea indeed. Otherwise, I'm not so sure.
* The current IOSchedulerSimple is still great for spinning disks, but
it makes a bunch
of assumptions about bandwidth that aren't accurate and should
probably require some
additional context from the driver. It seems advantageous to
consider using a heuristic,
measuring the bandwidth each round and using that to decide how
large of a batch
should be used for the next round.
Indeed. This is exactly what I did in the block_cache, which also had
a pretty dramatic latency reduction effect (10-200x.) So adding it
here likely makes a lot of sense.
I know you all are more focussed on stability and resolving beta 2
blockers, but this particular thing caught my interest so I spent some
time on it. I'm interested in some feedback on whether you think this
is worth pursuing and whether you think this kind of work is a good
idea right now. If there is interest in this, then I'll continue
experimenting and put together a proposal for review and then a patch.
It is certainly not a priority right now (and, as per the above
ticket, making changes down here is more dangerous than you may expect
because it might uncover nasty bugs elsewhere.) But certainly we need
this kind of work to be done, yes, even if I (personally) think other
things are more pressing...
-waddlesplash
Other related posts:
