Deferred work in net80211
Overview
The work of driving the driver, interface and node state machines is partially implemented as a set of deferred work tasks which are serialised on a driver task queue. This way the order of control plane operations can be guaranteed and work can be done without complicated lock ordering strategies.
Implementation
The current net80211 implementation uses FreeBSD taskqueues to provide a place for both net80211 and driver specific state machine tasks to be serialised into and run. This replaced the bulk of per-driver taskqueues for state management.
Each (struct ieee80211com) has an entry (ic_tq) which represents the state taskqueue. The FreeBSD implementation of taskqueues requires the caller create, initialise and add their own task to the queue.
net80211 and driver API
net80211 and drivers have the following API:
Initialising tasks currently just uses the FreeBSD taskqueue macros:
struct task - the FreeBSD taskqueue work item
TASK_INIT() - initialise a task with state pointer and callback
Work is handled via two calls:
ieee80211_runtask() - will schedule the given task to run
ieee80211_draintask() - will wait for the given task to complete if scheduled
Tasks are run in their order they are scheduled.
In addition, the following functions leverage taskqueues to provide known good states for certain control plane operations such as suspend, resume, interface stop, etc:
ieee80211_waitfor_parent() - will block the taskqueue and then wait for (some) pending work to complete.
Other parts of driver/net80211 code currently calls the taskqueue_* routines directly on the ic_tq rather than a platform API to abstract it.
Why use this versus mutexes and state variables?
net80211 has to handle a variety of state changes from a variety of sources. Here are some examples:
userland - (ioctl calls from hostapd/wpa_supplicant, ifconfig, other tools);
timers - eg a BAR timeout causing A-MPDU TX state to be torn out, nodes expiring, association / authentication timeouts;
transmit errors
received frames - plenty of 802.11 state changes based on received frames!
driver input - the driver / firmware itself may trigger state changes due to packet errors, firmware command success/failure, notifications about node timeouts, and much more.
These could all be implemented by holding mutexes whilst state changes occur, but in a lot of cases there may be other mutexes being held inside net80211, the ioctl layer, the driver stack, the upper layers of the network stack and .. well, a variety of other places. This also can lead into situations where the driver and net80211 end up calling into each other in circles just to get work done.
Here's an example - notification of an 802.11n channel width via a call to ieee80211_update_chw().
This happens when an 802.11 IE is received which indicates the channel width should change.
This will end up calling into the 802.11 stack to signal the channel width change.
It will also need to call back into the driver to potentially change the currently configured channel width.
If this were done without a deferred task, the flow would be driver -> net80211 -> driver (and then potentially -> net80211 again.)
Instead, deferring the work addresses a few things:
Any locks held across the driver receive path don't matter here, as the serialising is done via the task queue order, not by mutexes being held;
The work is serialised based on the order of received state changes (ie receving packets A, B and C which cause state changes A, B and C to be scheduled should result in A, B and C happening in that order);
There's no recursion from driver -> net80211 -> driver, or net80211 -> driver -> net80211 (except for some macros/utility calls.)
VAP state change handling / ieee80211_new_state()
The VAP newstate handling is an example of where the current task API falls short and it would benefit from being more dynamic.
The older net80211 code had a single task for newstate. Each call to ieee80211_new_state() to change the VAP state (AUTH, RUN, INIT, etc) would attempt to update the VAP state via a deferred task. However if multiple state changes came in quickly, the requested new state and argument would end up replacing the existing queued one, and the driver would not see the intermediate state changes.
This is fine for some - eg back to back channel width changes can be coalesced into one - but others such as the VAP state machine should not!
This changed in FreeBSD-14 / FreeBSD-15 to leverage an array of newstates to attempt to deal with this happening. ieee80211_new_state_locked() would request a free slot, and then ieee80211_newstate_cb() would get the next pending state from the list to handle.
Deferred tasks versus data path and control path
There is no implicit synchronisation between deferred tasks, the control path and the data path. Deferred tasks can and will run in parallel with packet transmit and receive and with ioctl / other control paths.
The goals of task deferral is to serialise these tasks between itself and other tasks. This has the side effect of happening outside of all the locking that may be occuring if it were done inline, but it does not preclude tasks themselves from having to use locks to serialise with the data/control paths.
Thus, data path, control path and deferred tasks must still use the appropriate mutexes to protect any state changes (global, vap, node, etc.) In most cases it'll be the driver lock (via IEEE80211_LOCK() ), but it may also involve the node table lock, power save queue lock, etc.
net80211 driver tasks
The following are a list of tasks which are global to the device and are defined in struct ieee80211com .
ic_parent_task - deferred parent processing
ic_promisc_task - deferred promiscuous configuration change processing
ic_mcast_task - deferred multicast config/filter processing
ic_chan_task - deferred global device channel change
ic_bmiss_task - deferred beacon miss handler
ic_chw_task - deferred HT channel width (20/40MHz) update processing
ic_restart_task - deferred device restart
The following are a list of tasks which are per-VAP and are defined in struct ieee80211vap .
iv_swbmiss_task - deferred per-vap beacon miss processing
iv_nstate_task (with iv_nstates, iv_nstate_args) - VAP state transition handling tasks
iv_wme_task - deferred WME (QoS configuration) update
iv_slot_task - deferred slot time update
iv_erp_protmode_task - deferred ERP/11g protection mode update
iv_preamble_task - deferred 802.11b preamble update
iv_ht_protmode_update - deferred 802.11n protection mode update
Future work
There's currently no way to schedule multiple instances of a task with some state into the taskqueue. Eg, the newstate task used to "miss" state changes; it now works around this by having an array of newstate task entries.
The task API is very much a thin wrapper around FreeBSD's taskqueue API. It really should become a platform API which is defined and implemented in ieee80211_freebsd.[ch].
Tasks are fire and forget. There's currently no way for the submitter to be called when thas task runs or is canceled; it can only run and then block via calling drain until its called or cancelled.
There's currently no way to suspend transmit and receive handling around state processing. This is not always desirable for packet performance and latency reasons but there are cases where this would be desirable (eg a channel width change would benefit from stopping and queueing transmit, waiting for the driver to finish transmitting, then change the channel width and then unblock transmit to continue.)