1 .. SPDX-License-Identifier: GPL-2.0
7 .. kernel-doc:: include/net/page_pool/helpers.h
8 :doc: page_pool allocator
23 +--------------------------------------------+
25 +--------------------------------------------+
28 | Pool empty | Pool has entries
31 +-----------------------+ +------------------------+
32 | alloc (and map) pages | | get page from cache |
33 +-----------------------+ +------------------------+
36 | cache available | No entries, refill
40 +-----------------+ +------------------+
41 | Fast cache | | ptr-ring cache |
42 +-----------------+ +------------------+
46 The number of pools created **must** match the number of hardware queues
47 unless hardware restrictions make that impossible. This would otherwise beat the
48 purpose of page pool, which is allocate pages fast from cache without locking.
49 This lockless guarantee naturally comes from running under a NAPI softirq.
50 The protection doesn't strictly have to be NAPI, any guarantee that allocating
51 a page will cause no race conditions is enough.
53 .. kernel-doc:: net/core/page_pool.c
54 :identifiers: page_pool_create
56 .. kernel-doc:: include/net/page_pool/types.h
57 :identifiers: struct page_pool_params
59 .. kernel-doc:: include/net/page_pool/helpers.h
60 :identifiers: page_pool_put_page page_pool_put_full_page
61 page_pool_recycle_direct page_pool_free_va
62 page_pool_dev_alloc_pages page_pool_dev_alloc_frag
63 page_pool_dev_alloc page_pool_dev_alloc_va
64 page_pool_get_dma_addr page_pool_get_dma_dir
66 .. kernel-doc:: net/core/page_pool.c
67 :identifiers: page_pool_put_page_bulk page_pool_get_stats
71 Driver is always responsible for syncing the pages for the CPU.
72 Drivers may choose to take care of syncing for the device as well
73 or set the ``PP_FLAG_DMA_SYNC_DEV`` flag to request that pages
74 allocated from the page pool are already synced for the device.
76 If ``PP_FLAG_DMA_SYNC_DEV`` is set, the driver must inform the core what portion
77 of the buffer has to be synced. This allows the core to avoid syncing the entire
78 page when the drivers knows that the device only accessed a portion of the page.
80 Most drivers will reserve headroom in front of the frame. This part
81 of the buffer is not touched by the device, so to avoid syncing
82 it drivers can set the ``offset`` field in struct page_pool_params
85 For pages recycled on the XDP xmit and skb paths the page pool will
86 use the ``max_len`` member of struct page_pool_params to decide how
87 much of the page needs to be synced (starting at ``offset``).
88 When directly freeing pages in the driver (page_pool_put_page())
89 the ``dma_sync_size`` argument specifies how much of the buffer needs
92 If in doubt set ``offset`` to 0, ``max_len`` to ``PAGE_SIZE`` and
93 pass -1 as ``dma_sync_size``. That combination of arguments is always
96 Note that the syncing parameters are for the entire page.
97 This is important to remember when using fragments (``PP_FLAG_PAGE_FRAG``),
98 where allocated buffers may be smaller than a full page.
99 Unless the driver author really understands page pool internals
100 it's recommended to always use ``offset = 0``, ``max_len = PAGE_SIZE``
101 with fragmented page pools.
103 Stats API and structures
104 ------------------------
105 If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
106 page_pool_get_stats() and structures described below are available.
107 It takes a pointer to a ``struct page_pool`` and a pointer to a struct
108 page_pool_stats allocated by the caller.
110 The API will fill in the provided struct page_pool_stats with
111 statistics about the page_pool.
113 .. kernel-doc:: include/net/page_pool/types.h
114 :identifiers: struct page_pool_recycle_stats
115 struct page_pool_alloc_stats
116 struct page_pool_stats
126 /* Page pool registration */
127 struct page_pool_params pp_params = { 0 };
128 struct xdp_rxq_info xdp_rxq;
132 /* internal DMA mapping in page_pool */
133 pp_params.flags = PP_FLAG_DMA_MAP;
134 pp_params.pool_size = DESC_NUM;
135 pp_params.nid = NUMA_NO_NODE;
136 pp_params.dev = priv->dev;
137 pp_params.napi = napi; /* only if locking is tied to NAPI */
138 pp_params.dma_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
139 page_pool = page_pool_create(&pp_params);
141 err = xdp_rxq_info_reg(&xdp_rxq, ndev, 0);
145 err = xdp_rxq_info_reg_mem_model(&xdp_rxq, MEM_TYPE_PAGE_POOL, page_pool);
156 enum dma_data_direction dma_dir;
158 dma_dir = page_pool_get_dma_dir(dring->page_pool);
159 while (done < budget) {
161 page_pool_recycle_direct(page_pool, page);
164 page_pool_recycle_direct(page_pool, page);
165 } else (packet_is_skb) {
166 skb_mark_for_recycle(skb);
167 new_page = page_pool_dev_alloc_pages(page_pool);
176 #ifdef CONFIG_PAGE_POOL_STATS
178 struct page_pool_stats stats = { 0 };
179 if (page_pool_get_stats(page_pool, &stats)) {
180 /* perhaps the driver reports statistics with ethool */
181 ethtool_print_allocation_stats(&stats.alloc_stats);
182 ethtool_print_recycle_stats(&stats.recycle_stats);
192 page_pool_put_full_page(page_pool, page, false);
193 xdp_rxq_info_unreg(&xdp_rxq);