--- /dev/null
+===============
+DRM Driver uAPI
+===============
+
+drm/i915 uAPI
+=============
+
+.. kernel-doc:: include/uapi/drm/i915_drm.h
drm-kms
drm-kms-helpers
drm-uapi
+ driver-uapi
drm-client
drivers
backlight
--- /dev/null
+=========================
+I915 DG1/LMEM RFC Section
+=========================
+
+Upstream plan
+=============
+For upstream the overall plan for landing all the DG1 stuff and turning it for
+real, with all the uAPI bits is:
+
+* Merge basic HW enabling of DG1(still without pciid)
+* Merge the uAPI bits behind special CONFIG_BROKEN(or so) flag
+ * At this point we can still make changes, but importantly this lets us
+ start running IGTs which can utilize local-memory in CI
+* Convert over to TTM, make sure it all keeps working. Some of the work items:
+ * TTM shrinker for discrete
+ * dma_resv_lockitem for full dma_resv_lock, i.e not just trylock
+ * Use TTM CPU pagefault handler
+ * Route shmem backend over to TTM SYSTEM for discrete
+ * TTM purgeable object support
+ * Move i915 buddy allocator over to TTM
+ * MMAP ioctl mode(see `I915 MMAP`_)
+ * SET/GET ioctl caching(see `I915 SET/GET CACHING`_)
+* Send RFC(with mesa-dev on cc) for final sign off on the uAPI
+* Add pciid for DG1 and turn on uAPI for real
+
+New object placement and region query uAPI
+==========================================
+Starting from DG1 we need to give userspace the ability to allocate buffers from
+device local-memory. Currently the driver supports gem_create, which can place
+buffers in system memory via shmem, and the usual assortment of other
+interfaces, like dumb buffers and userptr.
+
+To support this new capability, while also providing a uAPI which will work
+beyond just DG1, we propose to offer three new bits of uAPI:
+
+DRM_I915_QUERY_MEMORY_REGIONS
+-----------------------------
+New query ID which allows userspace to discover the list of supported memory
+regions(like system-memory and local-memory) for a given device. We identify
+each region with a class and instance pair, which should be unique. The class
+here would be DEVICE or SYSTEM, and the instance would be zero, on platforms
+like DG1.
+
+Side note: The class/instance design is borrowed from our existing engine uAPI,
+where we describe every physical engine in terms of its class, and the
+particular instance, since we can have more than one per class.
+
+In the future we also want to expose more information which can further
+describe the capabilities of a region.
+
+.. kernel-doc:: include/uapi/drm/i915_drm.h
+ :functions: drm_i915_gem_memory_class drm_i915_gem_memory_class_instance drm_i915_memory_region_info drm_i915_query_memory_regions
+
+GEM_CREATE_EXT
+--------------
+New ioctl which is basically just gem_create but now allows userspace to provide
+a chain of possible extensions. Note that if we don't provide any extensions and
+set flags=0 then we get the exact same behaviour as gem_create.
+
+Side note: We also need to support PXP[1] in the near future, which is also
+applicable to integrated platforms, and adds its own gem_create_ext extension,
+which basically lets userspace mark a buffer as "protected".
+
+.. kernel-doc:: include/uapi/drm/i915_drm.h
+ :functions: drm_i915_gem_create_ext
+
+I915_GEM_CREATE_EXT_MEMORY_REGIONS
+----------------------------------
+Implemented as an extension for gem_create_ext, we would now allow userspace to
+optionally provide an immutable list of preferred placements at creation time,
+in priority order, for a given buffer object. For the placements we expect
+them each to use the class/instance encoding, as per the output of the regions
+query. Having the list in priority order will be useful in the future when
+placing an object, say during eviction.
+
+.. kernel-doc:: include/uapi/drm/i915_drm.h
+ :functions: drm_i915_gem_create_ext_memory_regions
+
+One fair criticism here is that this seems a little over-engineered[2]. If we
+just consider DG1 then yes, a simple gem_create.flags or something is totally
+all that's needed to tell the kernel to allocate the buffer in local-memory or
+whatever. However looking to the future we need uAPI which can also support
+upcoming Xe HP multi-tile architecture in a sane way, where there can be
+multiple local-memory instances for a given device, and so using both class and
+instance in our uAPI to describe regions is desirable, although specifically
+for DG1 it's uninteresting, since we only have a single local-memory instance.
+
+Existing uAPI issues
+====================
+Some potential issues we still need to resolve.
+
+I915 MMAP
+---------
+In i915 there are multiple ways to MMAP GEM object, including mapping the same
+object using different mapping types(WC vs WB), i.e multiple active mmaps per
+object. TTM expects one MMAP at most for the lifetime of the object. If it
+turns out that we have to backpedal here, there might be some potential
+userspace fallout.
+
+I915 SET/GET CACHING
+--------------------
+In i915 we have set/get_caching ioctl. TTM doesn't let us to change this, but
+DG1 doesn't support non-snooped pcie transactions, so we can just always
+allocate as WB for smem-only buffers. If/when our hw gains support for
+non-snooped pcie transactions then we must fix this mode at allocation time as
+a new GEM extension.
+
+This is related to the mmap problem, because in general (meaning, when we're
+not running on intel cpus) the cpu mmap must not, ever, be inconsistent with
+allocation mode.
+
+Possible idea is to let the kernel picks the mmap mode for userspace from the
+following table:
+
+smem-only: WB. Userspace does not need to call clflush.
+
+smem+lmem: We only ever allow a single mode, so simply allocate this as uncached
+memory, and always give userspace a WC mapping. GPU still does snooped access
+here(assuming we can't turn it off like on DG1), which is a bit inefficient.
+
+lmem only: always WC
+
+This means on discrete you only get a single mmap mode, all others must be
+rejected. That's probably going to be a new default mode or something like
+that.
+
+Links
+=====
+[1] https://patchwork.freedesktop.org/series/86798/
+
+[2] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5599#note_553791
* Once the code has landed move all the documentation to the right places in
the main core, helper or driver sections.
+
+.. toctree::
+
+ i915_gem_lmem.rst
struct drm_framebuffer *fb;
struct drm_i915_gem_object *obj;
struct drm_mode_fb_cmd2 mode_cmd = *user_mode_cmd;
+ struct drm_i915_private *i915;
obj = i915_gem_object_lookup(filp, mode_cmd.handles[0]);
if (!obj)
return ERR_PTR(-ENOENT);
+ /* object is backed with LMEM for discrete */
+ i915 = to_i915(obj->base.dev);
+ if (HAS_LMEM(i915) && !i915_gem_object_is_lmem(obj)) {
+ /* object is "remote", not in local memory */
+ i915_gem_object_put(obj);
+ return ERR_PTR(-EREMOTE);
+ }
+
fb = intel_framebuffer_create(obj, &mode_cmd);
i915_gem_object_put(obj);
#include <drm/drm_fb_helper.h>
#include <drm/drm_fourcc.h>
+#include "gem/i915_gem_lmem.h"
+
#include "i915_drv.h"
#include "intel_display_types.h"
#include "intel_fbdev.h"
size = mode_cmd.pitches[0] * mode_cmd.height;
size = PAGE_ALIGN(size);
- /* If the FB is too big, just don't use it since fbdev is not very
- * important and we should probably use that space with FBC or other
- * features. */
obj = ERR_PTR(-ENODEV);
- if (size * 2 < dev_priv->stolen_usable_size)
- obj = i915_gem_object_create_stolen(dev_priv, size);
- if (IS_ERR(obj))
- obj = i915_gem_object_create_shmem(dev_priv, size);
+ if (HAS_LMEM(dev_priv)) {
+ obj = i915_gem_object_create_lmem(dev_priv, size,
+ I915_BO_ALLOC_CONTIGUOUS);
+ } else {
+ /*
+ * If the FB is too big, just don't use it since fbdev is not very
+ * important and we should probably use that space with FBC or other
+ * features.
+ */
+ if (size * 2 < dev_priv->stolen_usable_size)
+ obj = i915_gem_object_create_stolen(dev_priv, size);
+ if (IS_ERR(obj))
+ obj = i915_gem_object_create_shmem(dev_priv, size);
+ }
+
if (IS_ERR(obj)) {
drm_err(&dev_priv->drm, "failed to allocate framebuffer\n");
return PTR_ERR(obj);
unsigned long flags = 0;
bool prealloc = false;
void __iomem *vaddr;
+ struct drm_i915_gem_object *obj;
int ret;
if (intel_fb &&
info->fbops = &intelfb_ops;
/* setup aperture base/size for vesafb takeover */
- info->apertures->ranges[0].base = ggtt->gmadr.start;
- info->apertures->ranges[0].size = ggtt->mappable_end;
+ obj = intel_fb_obj(&intel_fb->base);
+ if (i915_gem_object_is_lmem(obj)) {
+ struct intel_memory_region *mem = obj->mm.region;
+
+ info->apertures->ranges[0].base = mem->io_start;
+ info->apertures->ranges[0].size = mem->total;
+
+ /* Use fbdev's framebuffer from lmem for discrete */
+ info->fix.smem_start =
+ (unsigned long)(mem->io_start +
+ i915_gem_object_get_dma_address(obj, 0));
+ info->fix.smem_len = obj->base.size;
+ } else {
+ info->apertures->ranges[0].base = ggtt->gmadr.start;
+ info->apertures->ranges[0].size = ggtt->mappable_end;
- /* Our framebuffer is the entirety of fbdev's system memory */
- info->fix.smem_start =
- (unsigned long)(ggtt->gmadr.start + vma->node.start);
- info->fix.smem_len = vma->node.size;
+ /* Our framebuffer is the entirety of fbdev's system memory */
+ info->fix.smem_start =
+ (unsigned long)(ggtt->gmadr.start + vma->node.start);
+ info->fix.smem_len = vma->node.size;
+ }
vaddr = i915_vma_pin_iomap(vma);
if (IS_ERR(vaddr)) {
return 0;
}
-__i915_active_call
static void frontbuffer_retire(struct i915_active *ref)
{
struct intel_frontbuffer *front =
atomic_set(&front->bits, 0);
i915_active_init(&front->write,
frontbuffer_active,
- i915_active_may_sleep(frontbuffer_retire));
+ frontbuffer_retire,
+ I915_ACTIVE_RETIRE_SLEEPS);
spin_lock(&i915->fb_tracking.lock);
if (rcu_access_pointer(obj->frontbuffer)) {
i830_overlay_clock_gating(dev_priv, true);
}
-__i915_active_call static void
-intel_overlay_last_flip_retire(struct i915_active *active)
+static void intel_overlay_last_flip_retire(struct i915_active *active)
{
struct intel_overlay *overlay =
container_of(active, typeof(*overlay), last_flip);
overlay->saturation = 146;
i915_active_init(&overlay->last_flip,
- NULL, intel_overlay_last_flip_retire);
+ NULL, intel_overlay_last_flip_retire, 0);
ret = get_registers(overlay, OVERLAY_NEEDS_PHYSICAL(dev_priv));
if (ret)
void *data;
};
-__i915_active_call
static void cb_retire(struct i915_active *base)
{
struct context_barrier_task *cb = container_of(base, typeof(*cb), base);
if (!cb)
return -ENOMEM;
- i915_active_init(&cb->base, NULL, cb_retire);
+ i915_active_init(&cb->base, NULL, cb_retire, 0);
err = i915_active_acquire(&cb->base);
if (err) {
kfree(cb);
*/
#include "gem/i915_gem_ioctls.h"
+#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
#include "i915_drv.h"
+#include "i915_trace.h"
+#include "i915_user_extensions.h"
+
+static u32 object_max_page_size(struct drm_i915_gem_object *obj)
+{
+ u32 max_page_size = 0;
+ int i;
+
+ for (i = 0; i < obj->mm.n_placements; i++) {
+ struct intel_memory_region *mr = obj->mm.placements[i];
+
+ GEM_BUG_ON(!is_power_of_2(mr->min_page_size));
+ max_page_size = max_t(u32, max_page_size, mr->min_page_size);
+ }
+
+ GEM_BUG_ON(!max_page_size);
+ return max_page_size;
+}
+
+static void object_set_placements(struct drm_i915_gem_object *obj,
+ struct intel_memory_region **placements,
+ unsigned int n_placements)
+{
+ GEM_BUG_ON(!n_placements);
+
+ /*
+ * For the common case of one memory region, skip storing an
+ * allocated array and just point at the region directly.
+ */
+ if (n_placements == 1) {
+ struct intel_memory_region *mr = placements[0];
+ struct drm_i915_private *i915 = mr->i915;
+
+ obj->mm.placements = &i915->mm.regions[mr->id];
+ obj->mm.n_placements = 1;
+ } else {
+ obj->mm.placements = placements;
+ obj->mm.n_placements = n_placements;
+ }
+}
+
+static int i915_gem_publish(struct drm_i915_gem_object *obj,
+ struct drm_file *file,
+ u64 *size_p,
+ u32 *handle_p)
+{
+ u64 size = obj->base.size;
+ int ret;
+
+ ret = drm_gem_handle_create(file, &obj->base, handle_p);
+ /* drop reference from allocate - handle holds it now */
+ i915_gem_object_put(obj);
+ if (ret)
+ return ret;
+
+ *size_p = size;
+ return 0;
+}
static int
-i915_gem_create(struct drm_file *file,
- struct intel_memory_region *mr,
- u64 *size_p,
- u32 *handle_p)
+i915_gem_setup(struct drm_i915_gem_object *obj, u64 size)
{
- struct drm_i915_gem_object *obj;
- u32 handle;
- u64 size;
+ struct intel_memory_region *mr = obj->mm.placements[0];
+ unsigned int flags;
int ret;
- GEM_BUG_ON(!is_power_of_2(mr->min_page_size));
- size = round_up(*size_p, mr->min_page_size);
+ size = round_up(size, object_max_page_size(obj));
if (size == 0)
return -EINVAL;
/* For most of the ABI (e.g. mmap) we think in system pages */
GEM_BUG_ON(!IS_ALIGNED(size, PAGE_SIZE));
- /* Allocate the new object */
- obj = i915_gem_object_create_region(mr, size, 0);
- if (IS_ERR(obj))
- return PTR_ERR(obj);
+ if (i915_gem_object_size_2big(size))
+ return -E2BIG;
- GEM_BUG_ON(size != obj->base.size);
+ /*
+ * For now resort to CPU based clearing for device local-memory, in the
+ * near future this will use the blitter engine for accelerated, GPU
+ * based clearing.
+ */
+ flags = 0;
+ if (mr->type == INTEL_MEMORY_LOCAL)
+ flags = I915_BO_ALLOC_CPU_CLEAR;
- ret = drm_gem_handle_create(file, &obj->base, &handle);
- /* drop reference from allocate - handle holds it now */
- i915_gem_object_put(obj);
+ ret = mr->ops->init_object(mr, obj, size, flags);
if (ret)
return ret;
- *handle_p = handle;
- *size_p = size;
+ GEM_BUG_ON(size != obj->base.size);
+
+ trace_i915_gem_object_create(obj);
return 0;
}
struct drm_device *dev,
struct drm_mode_create_dumb *args)
{
+ struct drm_i915_gem_object *obj;
+ struct intel_memory_region *mr;
enum intel_memory_type mem_type;
int cpp = DIV_ROUND_UP(args->bpp, 8);
u32 format;
+ int ret;
switch (cpp) {
case 1:
if (HAS_LMEM(to_i915(dev)))
mem_type = INTEL_MEMORY_LOCAL;
- return i915_gem_create(file,
- intel_memory_region_by_type(to_i915(dev),
- mem_type),
- &args->size, &args->handle);
+ obj = i915_gem_object_alloc();
+ if (!obj)
+ return -ENOMEM;
+
+ mr = intel_memory_region_by_type(to_i915(dev), mem_type);
+ object_set_placements(obj, &mr, 1);
+
+ ret = i915_gem_setup(obj, args->size);
+ if (ret)
+ goto object_free;
+
+ return i915_gem_publish(obj, file, &args->size, &args->handle);
+
+object_free:
+ i915_gem_object_free(obj);
+ return ret;
}
/**
{
struct drm_i915_private *i915 = to_i915(dev);
struct drm_i915_gem_create *args = data;
+ struct drm_i915_gem_object *obj;
+ struct intel_memory_region *mr;
+ int ret;
i915_gem_flush_free_objects(i915);
- return i915_gem_create(file,
- intel_memory_region_by_type(i915,
- INTEL_MEMORY_SYSTEM),
- &args->size, &args->handle);
+ obj = i915_gem_object_alloc();
+ if (!obj)
+ return -ENOMEM;
+
+ mr = intel_memory_region_by_type(i915, INTEL_MEMORY_SYSTEM);
+ object_set_placements(obj, &mr, 1);
+
+ ret = i915_gem_setup(obj, args->size);
+ if (ret)
+ goto object_free;
+
+ return i915_gem_publish(obj, file, &args->size, &args->handle);
+
+object_free:
+ i915_gem_object_free(obj);
+ return ret;
+}
+
+struct create_ext {
+ struct drm_i915_private *i915;
+ struct drm_i915_gem_object *vanilla_object;
+};
+
+static void repr_placements(char *buf, size_t size,
+ struct intel_memory_region **placements,
+ int n_placements)
+{
+ int i;
+
+ buf[0] = '\0';
+
+ for (i = 0; i < n_placements; i++) {
+ struct intel_memory_region *mr = placements[i];
+ int r;
+
+ r = snprintf(buf, size, "\n %s -> { class: %d, inst: %d }",
+ mr->name, mr->type, mr->instance);
+ if (r >= size)
+ return;
+
+ buf += r;
+ size -= r;
+ }
+}
+
+static int set_placements(struct drm_i915_gem_create_ext_memory_regions *args,
+ struct create_ext *ext_data)
+{
+ struct drm_i915_private *i915 = ext_data->i915;
+ struct drm_i915_gem_memory_class_instance __user *uregions =
+ u64_to_user_ptr(args->regions);
+ struct drm_i915_gem_object *obj = ext_data->vanilla_object;
+ struct intel_memory_region **placements;
+ u32 mask;
+ int i, ret = 0;
+
+ if (args->pad) {
+ drm_dbg(&i915->drm, "pad should be zero\n");
+ ret = -EINVAL;
+ }
+
+ if (!args->num_regions) {
+ drm_dbg(&i915->drm, "num_regions is zero\n");
+ ret = -EINVAL;
+ }
+
+ if (args->num_regions > ARRAY_SIZE(i915->mm.regions)) {
+ drm_dbg(&i915->drm, "num_regions is too large\n");
+ ret = -EINVAL;
+ }
+
+ if (ret)
+ return ret;
+
+ placements = kmalloc_array(args->num_regions,
+ sizeof(struct intel_memory_region *),
+ GFP_KERNEL);
+ if (!placements)
+ return -ENOMEM;
+
+ mask = 0;
+ for (i = 0; i < args->num_regions; i++) {
+ struct drm_i915_gem_memory_class_instance region;
+ struct intel_memory_region *mr;
+
+ if (copy_from_user(®ion, uregions, sizeof(region))) {
+ ret = -EFAULT;
+ goto out_free;
+ }
+
+ mr = intel_memory_region_lookup(i915,
+ region.memory_class,
+ region.memory_instance);
+ if (!mr || mr->private) {
+ drm_dbg(&i915->drm, "Device is missing region { class: %d, inst: %d } at index = %d\n",
+ region.memory_class, region.memory_instance, i);
+ ret = -EINVAL;
+ goto out_dump;
+ }
+
+ if (mask & BIT(mr->id)) {
+ drm_dbg(&i915->drm, "Found duplicate placement %s -> { class: %d, inst: %d } at index = %d\n",
+ mr->name, region.memory_class,
+ region.memory_instance, i);
+ ret = -EINVAL;
+ goto out_dump;
+ }
+
+ placements[i] = mr;
+ mask |= BIT(mr->id);
+
+ ++uregions;
+ }
+
+ if (obj->mm.placements) {
+ ret = -EINVAL;
+ goto out_dump;
+ }
+
+ object_set_placements(obj, placements, args->num_regions);
+ if (args->num_regions == 1)
+ kfree(placements);
+
+ return 0;
+
+out_dump:
+ if (1) {
+ char buf[256];
+
+ if (obj->mm.placements) {
+ repr_placements(buf,
+ sizeof(buf),
+ obj->mm.placements,
+ obj->mm.n_placements);
+ drm_dbg(&i915->drm,
+ "Placements were already set in previous EXT. Existing placements: %s\n",
+ buf);
+ }
+
+ repr_placements(buf, sizeof(buf), placements, i);
+ drm_dbg(&i915->drm, "New placements(so far validated): %s\n", buf);
+ }
+
+out_free:
+ kfree(placements);
+ return ret;
+}
+
+static int ext_set_placements(struct i915_user_extension __user *base,
+ void *data)
+{
+ struct drm_i915_gem_create_ext_memory_regions ext;
+
+ if (!IS_ENABLED(CONFIG_DRM_I915_UNSTABLE_FAKE_LMEM))
+ return -ENODEV;
+
+ if (copy_from_user(&ext, base, sizeof(ext)))
+ return -EFAULT;
+
+ return set_placements(&ext, data);
+}
+
+static const i915_user_extension_fn create_extensions[] = {
+ [I915_GEM_CREATE_EXT_MEMORY_REGIONS] = ext_set_placements,
+};
+
+/**
+ * Creates a new mm object and returns a handle to it.
+ * @dev: drm device pointer
+ * @data: ioctl data blob
+ * @file: drm file pointer
+ */
+int
+i915_gem_create_ext_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file)
+{
+ struct drm_i915_private *i915 = to_i915(dev);
+ struct drm_i915_gem_create_ext *args = data;
+ struct create_ext ext_data = { .i915 = i915 };
+ struct intel_memory_region **placements_ext;
+ struct drm_i915_gem_object *obj;
+ int ret;
+
+ if (args->flags)
+ return -EINVAL;
+
+ i915_gem_flush_free_objects(i915);
+
+ obj = i915_gem_object_alloc();
+ if (!obj)
+ return -ENOMEM;
+
+ ext_data.vanilla_object = obj;
+ ret = i915_user_extensions(u64_to_user_ptr(args->extensions),
+ create_extensions,
+ ARRAY_SIZE(create_extensions),
+ &ext_data);
+ placements_ext = obj->mm.placements;
+ if (ret)
+ goto object_free;
+
+ if (!placements_ext) {
+ struct intel_memory_region *mr =
+ intel_memory_region_by_type(i915, INTEL_MEMORY_SYSTEM);
+
+ object_set_placements(obj, &mr, 1);
+ }
+
+ ret = i915_gem_setup(obj, args->size);
+ if (ret)
+ goto object_free;
+
+ return i915_gem_publish(obj, file, &args->size, &args->handle);
+
+object_free:
+ if (obj->mm.n_placements > 1)
+ kfree(placements_ext);
+ i915_gem_object_free(obj);
+ return ret;
}
struct drm_file *file);
int i915_gem_create_ioctl(struct drm_device *dev, void *data,
struct drm_file *file);
+int i915_gem_create_ext_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file);
int i915_gem_execbuffer2_ioctl(struct drm_device *dev, void *data,
struct drm_file *file);
int i915_gem_get_aperture_ioctl(struct drm_device *dev, void *data,
.release = i915_gem_object_release_memory_region,
};
+void __iomem *
+i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
+ unsigned long n,
+ unsigned long size)
+{
+ resource_size_t offset;
+
+ GEM_BUG_ON(!i915_gem_object_is_contiguous(obj));
+
+ offset = i915_gem_object_get_dma_address(obj, n);
+ offset -= obj->mm.region->region.start;
+
+ return io_mapping_map_wc(&obj->mm.region->iomap, offset, size);
+}
+
bool i915_gem_object_is_lmem(struct drm_i915_gem_object *obj)
{
- return obj->ops == &i915_gem_lmem_obj_ops;
+ struct intel_memory_region *mr = obj->mm.region;
+
+ return mr && (mr->type == INTEL_MEMORY_LOCAL ||
+ mr->type == INTEL_MEMORY_STOLEN_LOCAL);
}
struct drm_i915_gem_object *
extern const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops;
+void __iomem *
+i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
+ unsigned long n,
+ unsigned long size);
+
bool i915_gem_object_is_lmem(struct drm_i915_gem_object *obj);
struct drm_i915_gem_object *
if (obj->ops->release)
obj->ops->release(obj);
+ if (obj->mm.n_placements > 1)
+ kfree(obj->mm.placements);
+
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
cond_resched();
#define I915_BO_ALLOC_CONTIGUOUS BIT(0)
#define I915_BO_ALLOC_VOLATILE BIT(1)
#define I915_BO_ALLOC_STRUCT_PAGE BIT(2)
+#define I915_BO_ALLOC_CPU_CLEAR BIT(3)
#define I915_BO_ALLOC_FLAGS (I915_BO_ALLOC_CONTIGUOUS | \
I915_BO_ALLOC_VOLATILE | \
- I915_BO_ALLOC_STRUCT_PAGE)
-#define I915_BO_READONLY BIT(3)
-#define I915_TILING_QUIRK_BIT 4 /* unknown swizzling; do not release! */
+ I915_BO_ALLOC_STRUCT_PAGE | \
+ I915_BO_ALLOC_CPU_CLEAR)
+#define I915_BO_READONLY BIT(4)
+#define I915_TILING_QUIRK_BIT 5 /* unknown swizzling; do not release! */
/*
* Is the object to be mapped as read-only to the GPU
atomic_t pages_pin_count;
atomic_t shrink_pin;
+ /**
+ * Priority list of potential placements for this object.
+ */
+ struct intel_memory_region **placements;
+ int n_placements;
+
/**
* Memory region for this object.
*/
sg_mark_end(sg);
i915_sg_trim(st);
+ /* Intended for kernel internal use only */
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
+ struct scatterlist *sg;
+ unsigned long i;
+
+ for_each_sg(st->sgl, sg, st->nents, i) {
+ unsigned int length;
+ void __iomem *vaddr;
+ dma_addr_t daddr;
+
+ daddr = sg_dma_address(sg);
+ daddr -= mem->region.start;
+ length = sg_dma_len(sg);
+
+ vaddr = io_mapping_map_wc(&mem->iomap, daddr, length);
+ memset64((void __force *)vaddr, 0, length / sizeof(u64));
+ io_mapping_unmap(vaddr);
+ }
+
+ wmb();
+ }
+
__i915_gem_object_set_pages(obj, st, sg_page_sizes);
return 0;
}
static bool unsafe_drop_pages(struct drm_i915_gem_object *obj,
- unsigned long shrink)
+ unsigned long shrink, bool trylock_vm)
{
unsigned long flags;
flags = 0;
if (shrink & I915_SHRINK_ACTIVE)
- flags = I915_GEM_OBJECT_UNBIND_ACTIVE;
+ flags |= I915_GEM_OBJECT_UNBIND_ACTIVE;
if (!(shrink & I915_SHRINK_BOUND))
- flags = I915_GEM_OBJECT_UNBIND_TEST;
+ flags |= I915_GEM_OBJECT_UNBIND_TEST;
+ if (trylock_vm)
+ flags |= I915_GEM_OBJECT_UNBIND_VM_TRYLOCK;
if (i915_gem_object_unbind(obj, flags) == 0)
return true;
unsigned long scanned = 0;
int err;
+ /* CHV + VTD workaround use stop_machine(); need to trylock vm->mutex */
+ bool trylock_vm = !ww && intel_vm_no_concurrent_access_wa(i915);
+
trace_i915_gem_shrink(i915, target, shrink);
/*
spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
err = 0;
- if (unsafe_drop_pages(obj, shrink)) {
+ if (unsafe_drop_pages(obj, shrink, trylock_vm)) {
/* May arrive from get_pages on another bo */
if (!ww) {
if (!i915_gem_object_trylock(obj))
#include <drm/drm_mm.h>
#include <drm/i915_drm.h>
+#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
#include "i915_drv.h"
#include "i915_gem_stolen.h"
}
}
+ /*
+ * With stolen lmem, we don't need to check if the address range
+ * overlaps with the non-stolen system memory range, since lmem is local
+ * to the gpu.
+ */
+ if (HAS_LMEM(i915))
+ return 0;
+
/*
* Verify that nothing else uses this physical address. Stolen
* memory should be reserved by the BIOS and hidden from the
}
}
-static int i915_gem_init_stolen(struct drm_i915_private *i915)
+static int i915_gem_init_stolen(struct intel_memory_region *mem)
{
+ struct drm_i915_private *i915 = mem->i915;
struct intel_uncore *uncore = &i915->uncore;
resource_size_t reserved_base, stolen_top;
resource_size_t reserved_total, reserved_size;
return 0;
}
- if (resource_size(&intel_graphics_stolen_res) == 0)
+ if (resource_size(&mem->region) == 0)
return 0;
- i915->dsm = intel_graphics_stolen_res;
+ i915->dsm = mem->region;
if (i915_adjust_stolen(i915, &i915->dsm))
return 0;
{
static struct lock_class_key lock_class;
unsigned int cache_level;
+ unsigned int flags;
int err;
+ /*
+ * Stolen objects are always physically contiguous since we just
+ * allocate one big block underneath using the drm_mm range allocator.
+ */
+ flags = I915_BO_ALLOC_CONTIGUOUS;
+
drm_gem_private_object_init(&mem->i915->drm, &obj->base, stolen->size);
- i915_gem_object_init(obj, &i915_gem_object_stolen_ops, &lock_class, 0);
+ i915_gem_object_init(obj, &i915_gem_object_stolen_ops, &lock_class, flags);
obj->stolen = stolen;
obj->read_domains = I915_GEM_DOMAIN_CPU | I915_GEM_DOMAIN_GTT;
if (WARN_ON(!i915_gem_object_trylock(obj)))
return -EBUSY;
+ i915_gem_object_init_memory_region(obj, mem);
+
err = i915_gem_object_pin_pages(obj);
- if (!err)
- i915_gem_object_init_memory_region(obj, mem);
+ if (err)
+ i915_gem_object_release_memory_region(obj);
i915_gem_object_unlock(obj);
return err;
if (!stolen)
return -ENOMEM;
- ret = i915_gem_stolen_insert_node(i915, stolen, size, 4096);
+ ret = i915_gem_stolen_insert_node(i915, stolen, size,
+ mem->min_page_size);
if (ret)
goto err_free;
i915_gem_object_create_stolen(struct drm_i915_private *i915,
resource_size_t size)
{
- return i915_gem_object_create_region(i915->mm.regions[INTEL_REGION_STOLEN_SMEM],
- size, I915_BO_ALLOC_CONTIGUOUS);
+ return i915_gem_object_create_region(i915->mm.stolen_region, size, 0);
}
-static int init_stolen(struct intel_memory_region *mem)
+static int init_stolen_smem(struct intel_memory_region *mem)
{
- intel_memory_region_set_name(mem, "stolen");
-
/*
* Initialise stolen early so that we may reserve preallocated
* objects for the BIOS to KMS transition.
*/
- return i915_gem_init_stolen(mem->i915);
+ return i915_gem_init_stolen(mem);
}
-static void release_stolen(struct intel_memory_region *mem)
+static void release_stolen_smem(struct intel_memory_region *mem)
{
i915_gem_cleanup_stolen(mem->i915);
}
-static const struct intel_memory_region_ops i915_region_stolen_ops = {
- .init = init_stolen,
- .release = release_stolen,
+static const struct intel_memory_region_ops i915_region_stolen_smem_ops = {
+ .init = init_stolen_smem,
+ .release = release_stolen_smem,
.init_object = _i915_gem_object_stolen_init,
};
-struct intel_memory_region *i915_gem_stolen_setup(struct drm_i915_private *i915)
+static int init_stolen_lmem(struct intel_memory_region *mem)
{
- return intel_memory_region_create(i915,
- intel_graphics_stolen_res.start,
- resource_size(&intel_graphics_stolen_res),
- PAGE_SIZE, 0,
- &i915_region_stolen_ops);
+ int err;
+
+ if (GEM_WARN_ON(resource_size(&mem->region) == 0))
+ return -ENODEV;
+
+ if (!io_mapping_init_wc(&mem->iomap,
+ mem->io_start,
+ resource_size(&mem->region)))
+ return -EIO;
+
+ /*
+ * TODO: For stolen lmem we mostly just care about populating the dsm
+ * related bits and setting up the drm_mm allocator for the range.
+ * Perhaps split up i915_gem_init_stolen() for this.
+ */
+ err = i915_gem_init_stolen(mem);
+ if (err)
+ goto err_fini;
+
+ return 0;
+
+err_fini:
+ io_mapping_fini(&mem->iomap);
+ return err;
+}
+
+static void release_stolen_lmem(struct intel_memory_region *mem)
+{
+ io_mapping_fini(&mem->iomap);
+ i915_gem_cleanup_stolen(mem->i915);
+}
+
+static const struct intel_memory_region_ops i915_region_stolen_lmem_ops = {
+ .init = init_stolen_lmem,
+ .release = release_stolen_lmem,
+ .init_object = _i915_gem_object_stolen_init,
+};
+
+struct intel_memory_region *
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
+{
+ struct intel_uncore *uncore = &i915->uncore;
+ struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
+ struct intel_memory_region *mem;
+ resource_size_t io_start;
+ resource_size_t lmem_size;
+ u64 lmem_base;
+
+ lmem_base = intel_uncore_read64(uncore, GEN12_DSMBASE);
+ if (GEM_WARN_ON(lmem_base >= pci_resource_len(pdev, 2)))
+ return ERR_PTR(-ENODEV);
+
+ lmem_size = pci_resource_len(pdev, 2) - lmem_base;
+ io_start = pci_resource_start(pdev, 2) + lmem_base;
+
+ mem = intel_memory_region_create(i915, lmem_base, lmem_size,
+ I915_GTT_PAGE_SIZE_4K, io_start,
+ &i915_region_stolen_lmem_ops);
+ if (IS_ERR(mem))
+ return mem;
+
+ /*
+ * TODO: consider creating common helper to just print all the
+ * interesting stuff from intel_memory_region, which we can use for all
+ * our probed regions.
+ */
+
+ drm_dbg(&i915->drm, "Stolen Local memory IO start: %pa\n",
+ &mem->io_start);
+
+ intel_memory_region_set_name(mem, "stolen-local");
+
+ mem->private = true;
+
+ return mem;
+}
+
+struct intel_memory_region*
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
+{
+ struct intel_memory_region *mem;
+
+ mem = intel_memory_region_create(i915,
+ intel_graphics_stolen_res.start,
+ resource_size(&intel_graphics_stolen_res),
+ PAGE_SIZE, 0,
+ &i915_region_stolen_smem_ops);
+ if (IS_ERR(mem))
+ return mem;
+
+ intel_memory_region_set_name(mem, "stolen-system");
+
+ mem->private = true;
+
+ return mem;
}
struct drm_i915_gem_object *
resource_size_t stolen_offset,
resource_size_t size)
{
- struct intel_memory_region *mem = i915->mm.regions[INTEL_REGION_STOLEN_SMEM];
+ struct intel_memory_region *mem = i915->mm.stolen_region;
struct drm_i915_gem_object *obj;
struct drm_mm_node *stolen;
int ret;
/* KISS and expect everything to be page-aligned */
if (GEM_WARN_ON(size == 0) ||
- GEM_WARN_ON(!IS_ALIGNED(size, I915_GTT_PAGE_SIZE)) ||
- GEM_WARN_ON(!IS_ALIGNED(stolen_offset, I915_GTT_MIN_ALIGNMENT)))
+ GEM_WARN_ON(!IS_ALIGNED(size, mem->min_page_size)) ||
+ GEM_WARN_ON(!IS_ALIGNED(stolen_offset, mem->min_page_size)))
return ERR_PTR(-EINVAL);
stolen = kzalloc(sizeof(*stolen), GFP_KERNEL);
u64 end);
void i915_gem_stolen_remove_node(struct drm_i915_private *dev_priv,
struct drm_mm_node *node);
-struct intel_memory_region *i915_gem_stolen_setup(struct drm_i915_private *i915);
+struct intel_memory_region *i915_gem_stolen_smem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *i915_gem_stolen_lmem_setup(struct drm_i915_private *i915);
struct drm_i915_gem_object *
i915_gem_object_create_stolen(struct drm_i915_private *dev_priv,
resource_size_t size);
static int check_scratch_page(struct i915_gem_context *ctx, u32 *out)
{
struct i915_address_space *vm;
- struct page *page;
u32 *vaddr;
int err = 0;
if (!vm)
return -ENODEV;
- page = __px_page(vm->scratch[0]);
- if (!page) {
+ if (!vm->scratch[0]) {
pr_err("No scratch page!\n");
return -EINVAL;
}
- vaddr = kmap(page);
- if (!vaddr) {
- pr_err("No (mappable) scratch page!\n");
- return -EINVAL;
- }
+ vaddr = __px_vaddr(vm->scratch[0]);
memcpy(out, vaddr, sizeof(*out));
if (memchr_inv(vaddr, *out, PAGE_SIZE)) {
pr_err("Inconsistent initial state of scratch page!\n");
err = -EINVAL;
}
- kunmap(page);
return err;
}
return true;
}
+static void object_set_placements(struct drm_i915_gem_object *obj,
+ struct intel_memory_region **placements,
+ unsigned int n_placements)
+{
+ GEM_BUG_ON(!n_placements);
+
+ if (n_placements == 1) {
+ struct drm_i915_private *i915 = to_i915(obj->base.dev);
+ struct intel_memory_region *mr = placements[0];
+
+ obj->mm.placements = &i915->mm.regions[mr->id];
+ obj->mm.n_placements = 1;
+ } else {
+ obj->mm.placements = placements;
+ obj->mm.n_placements = n_placements;
+ }
+}
+
#define expand32(x) (((x) << 0) | ((x) << 8) | ((x) << 16) | ((x) << 24))
static int __igt_mmap(struct drm_i915_private *i915,
struct drm_i915_gem_object *obj,
if (IS_ERR(obj))
return PTR_ERR(obj);
+ object_set_placements(obj, &mr, 1);
+
err = __igt_mmap(i915, obj, I915_MMAP_TYPE_GTT);
if (err == 0)
err = __igt_mmap(i915, obj, I915_MMAP_TYPE_WC);
if (IS_ERR(obj))
return PTR_ERR(obj);
+ object_set_placements(obj, &mr, 1);
+
err = __igt_mmap_access(i915, obj, I915_MMAP_TYPE_GTT);
if (err == 0)
err = __igt_mmap_access(i915, obj, I915_MMAP_TYPE_WB);
if (IS_ERR(obj))
return PTR_ERR(obj);
+ object_set_placements(obj, &mr, 1);
+
err = __igt_mmap_gpu(i915, obj, I915_MMAP_TYPE_GTT);
if (err == 0)
err = __igt_mmap_gpu(i915, obj, I915_MMAP_TYPE_WC);
if (IS_ERR(obj))
return PTR_ERR(obj);
+ object_set_placements(obj, &mr, 1);
+
err = __igt_mmap_revoke(i915, obj, I915_MMAP_TYPE_GTT);
if (err == 0)
err = __igt_mmap_revoke(i915, obj, I915_MMAP_TYPE_WC);
* entries back to scratch.
*/
- vaddr = kmap_atomic_px(pt);
+ vaddr = px_vaddr(pt);
memset32(vaddr + pte, scratch_pte, count);
- kunmap_atomic(vaddr);
pte = 0;
}
GEM_BUG_ON(!pd->entry[act_pt]);
- vaddr = kmap_atomic_px(i915_pt_entry(pd, act_pt));
+ vaddr = px_vaddr(i915_pt_entry(pd, act_pt));
do {
GEM_BUG_ON(sg_dma_len(iter.sg) < I915_GTT_PAGE_SIZE);
vaddr[act_pte] = pte_encode | GEN6_PTE_ADDR_ENCODE(iter.dma);
}
if (++act_pte == GEN6_PTES) {
- kunmap_atomic(vaddr);
- vaddr = kmap_atomic_px(i915_pt_entry(pd, ++act_pt));
+ vaddr = px_vaddr(i915_pt_entry(pd, ++act_pt));
act_pte = 0;
}
} while (1);
- kunmap_atomic(vaddr);
vma->page_sizes.gtt = I915_GTT_PAGE_SIZE;
}
goto err_scratch0;
}
- ret = pin_pt_dma(vm, vm->scratch[1]);
+ ret = map_pt_dma(vm, vm->scratch[1]);
if (ret)
goto err_scratch1;
if (!vma)
return ERR_PTR(-ENOMEM);
- i915_active_init(&vma->active, NULL, NULL);
+ i915_active_init(&vma->active, NULL, NULL, 0);
kref_init(&vma->ref);
mutex_init(&vma->pages_mutex);
atomic_read(&pt->used));
GEM_BUG_ON(!count || count >= atomic_read(&pt->used));
- vaddr = kmap_atomic_px(pt);
+ vaddr = px_vaddr(pt);
memset64(vaddr + gen8_pd_index(start, 0),
vm->scratch[0]->encode,
count);
- kunmap_atomic(vaddr);
atomic_sub(count, &pt->used);
start += count;
gen8_pte_t *vaddr;
pd = i915_pd_entry(pdp, gen8_pd_index(idx, 2));
- vaddr = kmap_atomic_px(i915_pt_entry(pd, gen8_pd_index(idx, 1)));
+ vaddr = px_vaddr(i915_pt_entry(pd, gen8_pd_index(idx, 1)));
do {
GEM_BUG_ON(sg_dma_len(iter->sg) < I915_GTT_PAGE_SIZE);
vaddr[gen8_pd_index(idx, 0)] = pte_encode | iter->dma;
}
clflush_cache_range(vaddr, PAGE_SIZE);
- kunmap_atomic(vaddr);
- vaddr = kmap_atomic_px(i915_pt_entry(pd, gen8_pd_index(idx, 1)));
+ vaddr = px_vaddr(i915_pt_entry(pd, gen8_pd_index(idx, 1)));
}
} while (1);
clflush_cache_range(vaddr, PAGE_SIZE);
- kunmap_atomic(vaddr);
return idx;
}
encode |= GEN8_PDE_PS_2M;
page_size = I915_GTT_PAGE_SIZE_2M;
- vaddr = kmap_atomic_px(pd);
+ vaddr = px_vaddr(pd);
} else {
struct i915_page_table *pt =
i915_pt_entry(pd, __gen8_pte_index(start, 1));
rem >= (I915_PDES - index) * I915_GTT_PAGE_SIZE))
maybe_64K = __gen8_pte_index(start, 1);
- vaddr = kmap_atomic_px(pt);
+ vaddr = px_vaddr(pt);
}
do {
} while (rem >= page_size && index < I915_PDES);
clflush_cache_range(vaddr, PAGE_SIZE);
- kunmap_atomic(vaddr);
/*
* Is it safe to mark the 2M block as 64K? -- Either we have
!iter->sg && IS_ALIGNED(vma->node.start +
vma->node.size,
I915_GTT_PAGE_SIZE_2M)))) {
- vaddr = kmap_atomic_px(pd);
+ vaddr = px_vaddr(pd);
vaddr[maybe_64K] |= GEN8_PDE_IPS_64K;
- kunmap_atomic(vaddr);
page_size = I915_GTT_PAGE_SIZE_64K;
/*
u16 i;
encode = vma->vm->scratch[0]->encode;
- vaddr = kmap_atomic_px(i915_pt_entry(pd, maybe_64K));
+ vaddr = px_vaddr(i915_pt_entry(pd, maybe_64K));
for (i = 1; i < index; i += 16)
memset64(vaddr + i, encode, 15);
- kunmap_atomic(vaddr);
}
}
if (IS_ERR(obj))
goto free_scratch;
- ret = pin_pt_dma(vm, obj);
+ ret = map_pt_dma(vm, obj);
if (ret) {
i915_gem_object_put(obj);
goto free_scratch;
if (IS_ERR(pde))
return PTR_ERR(pde);
- err = pin_pt_dma(vm, pde->pt.base);
+ err = map_pt_dma(vm, pde->pt.base);
if (err) {
free_pd(vm, pde);
return err;
goto err_pd;
}
- err = pin_pt_dma(vm, pd->pt.base);
+ err = map_pt_dma(vm, pd->pt.base);
if (err)
goto err_pd;
*/
ppgtt->vm.has_read_only = !IS_GEN_RANGE(gt->i915, 11, 12);
- ppgtt->vm.alloc_pt_dma = alloc_pt_dma;
+ if (HAS_LMEM(gt->i915))
+ ppgtt->vm.alloc_pt_dma = alloc_pt_lmem;
+ else
+ ppgtt->vm.alloc_pt_dma = alloc_pt_dma;
err = gen8_init_scratch(&ppgtt->vm);
if (err)
intel_context_put(ce);
}
-__i915_active_call
static void __intel_context_retire(struct i915_active *active)
{
struct intel_context *ce = container_of(active, typeof(*ce), active);
mutex_init(&ce->pin_mutex);
i915_active_init(&ce->active,
- __intel_context_active, __intel_context_retire);
+ __intel_context_active, __intel_context_retire, 0);
}
void intel_context_fini(struct intel_context *ce)
#include "i915_reg.h"
#include "i915_request.h"
#include "i915_selftest.h"
-#include "gt/intel_timeline.h"
#include "intel_engine_types.h"
+#include "intel_gt_types.h"
+#include "intel_timeline.h"
#include "intel_workarounds.h"
struct drm_printer;
#define ENGINE_MOCK 1
#define ENGINE_VIRTUAL 2
+static inline bool intel_engine_uses_guc(const struct intel_engine_cs *engine)
+{
+ return engine->gt->submission_method >= INTEL_SUBMISSION_GUC;
+}
+
static inline bool
intel_engine_has_preempt_reset(const struct intel_engine_cs *engine)
{
intel_engine_set_hwsp_writemask(engine, ~0u);
}
+static void nop_irq_handler(struct intel_engine_cs *engine, u16 iir)
+{
+ GEM_DEBUG_WARN_ON(iir);
+}
+
static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id)
{
const struct engine_info *info = &intel_engines[id];
engine->hw_id = info->hw_id;
engine->guc_id = MAKE_GUC_ID(info->class, info->instance);
+ engine->irq_handler = nop_irq_handler;
+
engine->class = info->class;
engine->instance = info->instance;
__sprint_engine_name(engine);
return 0;
err_context:
- intel_context_put(ce);
+ destroy_pinned_context(ce);
return ret;
}
enum intel_engine_id id;
int err;
- if (intel_uc_uses_guc_submission(>->uc))
+ if (intel_uc_uses_guc_submission(>->uc)) {
+ gt->submission_method = INTEL_SUBMISSION_GUC;
setup = intel_guc_submission_setup;
- else if (HAS_EXECLISTS(gt->i915))
+ } else if (HAS_EXECLISTS(gt->i915)) {
+ gt->submission_method = INTEL_SUBMISSION_ELSP;
setup = intel_execlists_submission_setup;
- else
+ } else {
+ gt->submission_method = INTEL_SUBMISSION_RING;
setup = intel_ring_submission_setup;
+ }
for_each_engine(engine, gt, id) {
err = engine_setup_common(engine);
drm_printf(m, "\tIPEHR: 0x%08x\n", ENGINE_READ(engine, IPEHR));
}
- if (intel_engine_in_guc_submission_mode(engine)) {
+ if (intel_engine_uses_guc(engine)) {
/* nothing to print yet */
} else if (HAS_EXECLISTS(dev_priv)) {
struct i915_request * const *port, *rq;
if (ce->state) {
struct drm_i915_gem_object *obj = ce->state->obj;
- int type = i915_coherent_map_type(ce->engine->i915);
+ int type = i915_coherent_map_type(ce->engine->i915, obj, true);
void *map;
if (!i915_gem_object_trylock(obj))
u32 irq_enable_mask; /* bitmask to enable ring interrupt */
void (*irq_enable)(struct intel_engine_cs *engine);
void (*irq_disable)(struct intel_engine_cs *engine);
+ void (*irq_handler)(struct intel_engine_cs *engine, u16 iir);
void (*sanitize)(struct intel_engine_cs *engine);
int (*resume)(struct intel_engine_cs *engine);
#define I915_ENGINE_HAS_PREEMPTION BIT(2)
#define I915_ENGINE_HAS_SEMAPHORES BIT(3)
#define I915_ENGINE_HAS_TIMESLICES BIT(4)
-#define I915_ENGINE_NEEDS_BREADCRUMB_TASKLET BIT(5)
-#define I915_ENGINE_IS_VIRTUAL BIT(6)
-#define I915_ENGINE_HAS_RELATIVE_MMIO BIT(7)
-#define I915_ENGINE_REQUIRES_CMD_PARSER BIT(8)
+#define I915_ENGINE_IS_VIRTUAL BIT(5)
+#define I915_ENGINE_HAS_RELATIVE_MMIO BIT(6)
+#define I915_ENGINE_REQUIRES_CMD_PARSER BIT(7)
unsigned int flags;
/*
return engine->flags & I915_ENGINE_HAS_TIMESLICES;
}
-static inline bool
-intel_engine_needs_breadcrumb_tasklet(const struct intel_engine_cs *engine)
-{
- return engine->flags & I915_ENGINE_NEEDS_BREADCRUMB_TASKLET;
-}
-
static inline bool
intel_engine_is_virtual(const struct intel_engine_cs *engine)
{
#include "intel_engine_stats.h"
#include "intel_execlists_submission.h"
#include "intel_gt.h"
+#include "intel_gt_irq.h"
#include "intel_gt_pm.h"
#include "intel_gt_requests.h"
#include "intel_lrc.h"
*/
GEM_BUG_ON(!tasklet_is_locked(&execlists->tasklet) &&
!reset_in_progress(execlists));
- GEM_BUG_ON(!intel_engine_in_execlists_submission_mode(engine));
/*
* Note that csb_write, csb_status may be either in HWSP or mmio.
rcu_read_unlock();
}
+static void execlists_irq_handler(struct intel_engine_cs *engine, u16 iir)
+{
+ bool tasklet = false;
+
+ if (unlikely(iir & GT_CS_MASTER_ERROR_INTERRUPT)) {
+ u32 eir;
+
+ /* Upper 16b are the enabling mask, rsvd for internal errors */
+ eir = ENGINE_READ(engine, RING_EIR) & GENMASK(15, 0);
+ ENGINE_TRACE(engine, "CS error: %x\n", eir);
+
+ /* Disable the error interrupt until after the reset */
+ if (likely(eir)) {
+ ENGINE_WRITE(engine, RING_EMR, ~0u);
+ ENGINE_WRITE(engine, RING_EIR, eir);
+ WRITE_ONCE(engine->execlists.error_interrupt, eir);
+ tasklet = true;
+ }
+ }
+
+ if (iir & GT_WAIT_SEMAPHORE_INTERRUPT) {
+ WRITE_ONCE(engine->execlists.yield,
+ ENGINE_READ_FW(engine, RING_EXECLIST_STATUS_HI));
+ ENGINE_TRACE(engine, "semaphore yield: %08x\n",
+ engine->execlists.yield);
+ if (del_timer(&engine->execlists.timer))
+ tasklet = true;
+ }
+
+ if (iir & GT_CONTEXT_SWITCH_INTERRUPT)
+ tasklet = true;
+
+ if (iir & GT_RENDER_USER_INTERRUPT)
+ intel_engine_signal_breadcrumbs(engine);
+
+ if (tasklet)
+ tasklet_hi_schedule(&engine->execlists.tasklet);
+}
+
static void __execlists_kick(struct intel_engine_execlists *execlists)
{
/* Kick the tasklet for some interrupt coalescing and reset handling */
engine->submit_request = execlists_submit_request;
engine->schedule = i915_schedule;
engine->execlists.tasklet.callback = execlists_submission_tasklet;
-
- engine->reset.prepare = execlists_reset_prepare;
- engine->reset.rewind = execlists_reset_rewind;
- engine->reset.cancel = execlists_reset_cancel;
- engine->reset.finish = execlists_reset_finish;
-
- engine->park = execlists_park;
- engine->unpark = NULL;
-
- engine->flags |= I915_ENGINE_SUPPORTS_STATS;
- if (!intel_vgpu_active(engine->i915)) {
- engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
- if (can_preempt(engine)) {
- engine->flags |= I915_ENGINE_HAS_PREEMPTION;
- if (IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
- engine->flags |= I915_ENGINE_HAS_TIMESLICES;
- }
- }
-
- if (intel_engine_has_preemption(engine))
- engine->emit_bb_start = gen8_emit_bb_start;
- else
- engine->emit_bb_start = gen8_emit_bb_start_noarb;
}
static void execlists_shutdown(struct intel_engine_cs *engine)
engine->cops = &execlists_context_ops;
engine->request_alloc = execlists_request_alloc;
+ engine->reset.prepare = execlists_reset_prepare;
+ engine->reset.rewind = execlists_reset_rewind;
+ engine->reset.cancel = execlists_reset_cancel;
+ engine->reset.finish = execlists_reset_finish;
+
+ engine->park = execlists_park;
+ engine->unpark = NULL;
+
engine->emit_flush = gen8_emit_flush_xcs;
engine->emit_init_breadcrumb = gen8_emit_init_breadcrumb;
engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb_xcs;
* until a more refined solution exists.
*/
}
+ intel_engine_set_irq_handler(engine, execlists_irq_handler);
+
+ engine->flags |= I915_ENGINE_SUPPORTS_STATS;
+ if (!intel_vgpu_active(engine->i915)) {
+ engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
+ if (can_preempt(engine)) {
+ engine->flags |= I915_ENGINE_HAS_PREEMPTION;
+ if (IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
+ engine->flags |= I915_ENGINE_HAS_TIMESLICES;
+ }
+ }
+
+ if (intel_engine_has_preemption(engine))
+ engine->emit_bb_start = gen8_emit_bb_start;
+ else
+ engine->emit_bb_start = gen8_emit_bb_start_noarb;
}
static void logical_ring_default_irqs(struct intel_engine_cs *engine)
spin_unlock_irqrestore(&engine->active.lock, flags);
}
-bool
-intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine)
-{
- return engine->set_default_submission ==
- execlists_set_default_submission;
-}
-
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftest_execlists.c"
#endif
const struct intel_engine_cs *master,
const struct intel_engine_cs *sibling);
-bool
-intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine);
-
#endif /* __INTEL_EXECLISTS_SUBMISSION_H__ */
goto err_ppgtt;
i915_gem_object_lock(ppgtt->vm.scratch[0], NULL);
- err = i915_vm_pin_pt_stash(&ppgtt->vm, &stash);
+ err = i915_vm_map_pt_stash(&ppgtt->vm, &stash);
i915_gem_object_unlock(ppgtt->vm.scratch[0]);
if (err)
goto err_stash;
ggtt->vm.insert_entries = gen8_ggtt_insert_entries;
- /* Serialize GTT updates with aperture access on BXT if VT-d is on. */
- if (intel_ggtt_update_needs_vtd_wa(i915) ||
- IS_CHERRYVIEW(i915) /* fails with concurrent use/update */) {
+ /*
+ * Serialize GTT updates with aperture access on BXT if VT-d is on,
+ * and always on CHV.
+ */
+ if (intel_vm_no_concurrent_access_wa(i915)) {
ggtt->vm.insert_entries = bxt_vtd_ggtt_insert_entries__BKL;
ggtt->vm.insert_page = bxt_vtd_ggtt_insert_page__BKL;
ggtt->vm.bind_async_flags =
* banks of memory are paired and unswizzled on the
* uneven portion, so leave that as unknown.
*/
- if (intel_uncore_read16(uncore, C0DRB3) ==
- intel_uncore_read16(uncore, C1DRB3)) {
+ if (intel_uncore_read16(uncore, C0DRB3_BW) ==
+ intel_uncore_read16(uncore, C1DRB3_BW)) {
swizzle_x = I915_BIT_6_SWIZZLE_9_10;
swizzle_y = I915_BIT_6_SWIZZLE_9;
}
for (i = 0; i < num_fences; i++) {
struct i915_fence_reg *fence = &ggtt->fence_regs[i];
- i915_active_init(&fence->active, NULL, NULL);
+ i915_active_init(&fence->active, NULL, NULL, 0);
fence->ggtt = ggtt;
fence->id = i;
list_add_tail(&fence->link, &ggtt->fence_list);
round_jiffies_up_relative(HZ));
}
-__i915_active_call
static void pool_retire(struct i915_active *ref)
{
struct intel_gt_buffer_pool_node *node =
node->age = 0;
node->pool = pool;
node->pinned = false;
- i915_active_init(&node->active, NULL, pool_retire);
+ i915_active_init(&node->active, NULL, pool_retire, 0);
obj = i915_gem_object_create_internal(gt->i915, sz);
if (IS_ERR(obj)) {
intel_guc_to_host_event_handler(guc);
}
-static void
-cs_irq_handler(struct intel_engine_cs *engine, u32 iir)
-{
- bool tasklet = false;
-
- if (unlikely(iir & GT_CS_MASTER_ERROR_INTERRUPT)) {
- u32 eir;
-
- /* Upper 16b are the enabling mask, rsvd for internal errors */
- eir = ENGINE_READ(engine, RING_EIR) & GENMASK(15, 0);
- ENGINE_TRACE(engine, "CS error: %x\n", eir);
-
- /* Disable the error interrupt until after the reset */
- if (likely(eir)) {
- ENGINE_WRITE(engine, RING_EMR, ~0u);
- ENGINE_WRITE(engine, RING_EIR, eir);
- WRITE_ONCE(engine->execlists.error_interrupt, eir);
- tasklet = true;
- }
- }
-
- if (iir & GT_WAIT_SEMAPHORE_INTERRUPT) {
- WRITE_ONCE(engine->execlists.yield,
- ENGINE_READ_FW(engine, RING_EXECLIST_STATUS_HI));
- ENGINE_TRACE(engine, "semaphore yield: %08x\n",
- engine->execlists.yield);
- if (del_timer(&engine->execlists.timer))
- tasklet = true;
- }
-
- if (iir & GT_CONTEXT_SWITCH_INTERRUPT)
- tasklet = true;
-
- if (iir & GT_RENDER_USER_INTERRUPT) {
- intel_engine_signal_breadcrumbs(engine);
- tasklet |= intel_engine_needs_breadcrumb_tasklet(engine);
- }
-
- if (tasklet)
- tasklet_hi_schedule(&engine->execlists.tasklet);
-}
-
static u32
gen11_gt_engine_identity(struct intel_gt *gt,
const unsigned int bank, const unsigned int bit)
engine = NULL;
if (likely(engine))
- return cs_irq_handler(engine, iir);
+ return intel_engine_cs_irq(engine, iir);
WARN_ONCE(1, "unhandled engine interrupt class=0x%x, instance=0x%x\n",
class, instance);
void gen5_gt_irq_handler(struct intel_gt *gt, u32 gt_iir)
{
if (gt_iir & GT_RENDER_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(gt->engine_class[RENDER_CLASS][0]);
+ intel_engine_cs_irq(gt->engine_class[RENDER_CLASS][0],
+ gt_iir);
+
if (gt_iir & ILK_BSD_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(gt->engine_class[VIDEO_DECODE_CLASS][0]);
+ intel_engine_cs_irq(gt->engine_class[VIDEO_DECODE_CLASS][0],
+ gt_iir);
}
static void gen7_parity_error_irq_handler(struct intel_gt *gt, u32 iir)
void gen6_gt_irq_handler(struct intel_gt *gt, u32 gt_iir)
{
if (gt_iir & GT_RENDER_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(gt->engine_class[RENDER_CLASS][0]);
+ intel_engine_cs_irq(gt->engine_class[RENDER_CLASS][0],
+ gt_iir);
+
if (gt_iir & GT_BSD_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(gt->engine_class[VIDEO_DECODE_CLASS][0]);
+ intel_engine_cs_irq(gt->engine_class[VIDEO_DECODE_CLASS][0],
+ gt_iir >> 12);
+
if (gt_iir & GT_BLT_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(gt->engine_class[COPY_ENGINE_CLASS][0]);
+ intel_engine_cs_irq(gt->engine_class[COPY_ENGINE_CLASS][0],
+ gt_iir >> 22);
if (gt_iir & (GT_BLT_CS_ERROR_INTERRUPT |
GT_BSD_CS_ERROR_INTERRUPT |
if (master_ctl & (GEN8_GT_RCS_IRQ | GEN8_GT_BCS_IRQ)) {
iir = raw_reg_read(regs, GEN8_GT_IIR(0));
if (likely(iir)) {
- cs_irq_handler(gt->engine_class[RENDER_CLASS][0],
- iir >> GEN8_RCS_IRQ_SHIFT);
- cs_irq_handler(gt->engine_class[COPY_ENGINE_CLASS][0],
- iir >> GEN8_BCS_IRQ_SHIFT);
+ intel_engine_cs_irq(gt->engine_class[RENDER_CLASS][0],
+ iir >> GEN8_RCS_IRQ_SHIFT);
+ intel_engine_cs_irq(gt->engine_class[COPY_ENGINE_CLASS][0],
+ iir >> GEN8_BCS_IRQ_SHIFT);
raw_reg_write(regs, GEN8_GT_IIR(0), iir);
}
}
if (master_ctl & (GEN8_GT_VCS0_IRQ | GEN8_GT_VCS1_IRQ)) {
iir = raw_reg_read(regs, GEN8_GT_IIR(1));
if (likely(iir)) {
- cs_irq_handler(gt->engine_class[VIDEO_DECODE_CLASS][0],
- iir >> GEN8_VCS0_IRQ_SHIFT);
- cs_irq_handler(gt->engine_class[VIDEO_DECODE_CLASS][1],
- iir >> GEN8_VCS1_IRQ_SHIFT);
+ intel_engine_cs_irq(gt->engine_class[VIDEO_DECODE_CLASS][0],
+ iir >> GEN8_VCS0_IRQ_SHIFT);
+ intel_engine_cs_irq(gt->engine_class[VIDEO_DECODE_CLASS][1],
+ iir >> GEN8_VCS1_IRQ_SHIFT);
raw_reg_write(regs, GEN8_GT_IIR(1), iir);
}
}
if (master_ctl & GEN8_GT_VECS_IRQ) {
iir = raw_reg_read(regs, GEN8_GT_IIR(3));
if (likely(iir)) {
- cs_irq_handler(gt->engine_class[VIDEO_ENHANCEMENT_CLASS][0],
- iir >> GEN8_VECS_IRQ_SHIFT);
+ intel_engine_cs_irq(gt->engine_class[VIDEO_ENHANCEMENT_CLASS][0],
+ iir >> GEN8_VECS_IRQ_SHIFT);
raw_reg_write(regs, GEN8_GT_IIR(3), iir);
}
}
#include <linux/types.h>
+#include "intel_engine_types.h"
+
struct intel_gt;
#define GEN8_GT_IRQS (GEN8_GT_RCS_IRQ | \
void gen8_gt_irq_reset(struct intel_gt *gt);
void gen8_gt_irq_postinstall(struct intel_gt *gt);
+static inline void intel_engine_cs_irq(struct intel_engine_cs *engine, u16 iir)
+{
+ if (iir)
+ engine->irq_handler(engine, iir);
+}
+
+static inline void
+intel_engine_set_irq_handler(struct intel_engine_cs *engine,
+ void (*fn)(struct intel_engine_cs *engine,
+ u16 iir))
+{
+ /*
+ * As the interrupt is live as allocate and setup the engines,
+ * err on the side of caution and apply barriers to updating
+ * the irq handler callback. This assures that when we do use
+ * the engine, we will receive interrupts only to ourselves,
+ * and not lose any.
+ */
+ smp_store_mb(engine->irq_handler, fn);
+}
+
#endif /* INTEL_GT_IRQ_H */
struct intel_engine_cs;
struct intel_uncore;
+enum intel_submission_method {
+ INTEL_SUBMISSION_RING,
+ INTEL_SUBMISSION_ELSP,
+ INTEL_SUBMISSION_GUC,
+};
+
struct intel_gt {
struct drm_i915_private *i915;
struct intel_uncore *uncore;
struct intel_engine_cs *engine[I915_NUM_ENGINES];
struct intel_engine_cs *engine_class[MAX_ENGINE_CLASS + 1]
[MAX_ENGINE_INSTANCE + 1];
+ enum intel_submission_method submission_method;
/*
* Default address space (either GGTT or ppGTT depending on arch).
#include <linux/fault-inject.h>
+#include "gem/i915_gem_lmem.h"
#include "i915_trace.h"
#include "intel_gt.h"
#include "intel_gtt.h"
+struct drm_i915_gem_object *alloc_pt_lmem(struct i915_address_space *vm, int sz)
+{
+ struct drm_i915_gem_object *obj;
+
+ obj = i915_gem_object_create_lmem(vm->i915, sz, 0);
+ /*
+ * Ensure all paging structures for this vm share the same dma-resv
+ * object underneath, with the idea that one object_lock() will lock
+ * them all at once.
+ */
+ if (!IS_ERR(obj))
+ obj->base.resv = &vm->resv;
+ return obj;
+}
+
struct drm_i915_gem_object *alloc_pt_dma(struct i915_address_space *vm, int sz)
{
struct drm_i915_gem_object *obj;
i915_gem_shrink_all(vm->i915);
obj = i915_gem_object_create_internal(vm->i915, sz);
- /* ensure all dma objects have the same reservation class */
+ /*
+ * Ensure all paging structures for this vm share the same dma-resv
+ * object underneath, with the idea that one object_lock() will lock
+ * them all at once.
+ */
if (!IS_ERR(obj))
obj->base.resv = &vm->resv;
return obj;
}
-int pin_pt_dma(struct i915_address_space *vm, struct drm_i915_gem_object *obj)
+int map_pt_dma(struct i915_address_space *vm, struct drm_i915_gem_object *obj)
{
- int err;
+ enum i915_map_type type;
+ void *vaddr;
- i915_gem_object_lock(obj, NULL);
- err = i915_gem_object_pin_pages(obj);
- i915_gem_object_unlock(obj);
- if (err)
- return err;
+ type = i915_coherent_map_type(vm->i915, obj, true);
+ vaddr = i915_gem_object_pin_map_unlocked(obj, type);
+ if (IS_ERR(vaddr))
+ return PTR_ERR(vaddr);
i915_gem_object_make_unshrinkable(obj);
return 0;
}
-int pin_pt_dma_locked(struct i915_address_space *vm, struct drm_i915_gem_object *obj)
+int map_pt_dma_locked(struct i915_address_space *vm, struct drm_i915_gem_object *obj)
{
- int err;
+ enum i915_map_type type;
+ void *vaddr;
- err = i915_gem_object_pin_pages(obj);
- if (err)
- return err;
+ type = i915_coherent_map_type(vm->i915, obj, true);
+ vaddr = i915_gem_object_pin_map(obj, type);
+ if (IS_ERR(vaddr))
+ return PTR_ERR(vaddr);
i915_gem_object_make_unshrinkable(obj);
return 0;
*/
mutex_init(&vm->mutex);
lockdep_set_subclass(&vm->mutex, subclass);
- i915_gem_shrinker_taints_mutex(vm->i915, &vm->mutex);
+
+ if (!intel_vm_no_concurrent_access_wa(vm->i915)) {
+ i915_gem_shrinker_taints_mutex(vm->i915, &vm->mutex);
+ } else {
+ /*
+ * CHV + BXT VTD workaround use stop_machine(),
+ * which is allowed to allocate memory. This means &vm->mutex
+ * is the outer lock, and in theory we can allocate memory inside
+ * it through stop_machine().
+ *
+ * Add the annotation for this, we use trylock in shrinker.
+ */
+ mutex_acquire(&vm->mutex.dep_map, 0, 0, _THIS_IP_);
+ might_alloc(GFP_KERNEL);
+ mutex_release(&vm->mutex.dep_map, _THIS_IP_);
+ }
dma_resv_init(&vm->resv);
GEM_BUG_ON(!vm->total);
memset(&vma->page_sizes, 0, sizeof(vma->page_sizes));
}
+void *__px_vaddr(struct drm_i915_gem_object *p)
+{
+ enum i915_map_type type;
+
+ GEM_BUG_ON(!i915_gem_object_has_pages(p));
+ return page_unpack_bits(p->mm.mapping, &type);
+}
+
dma_addr_t __px_dma(struct drm_i915_gem_object *p)
{
GEM_BUG_ON(!i915_gem_object_has_pages(p));
void
fill_page_dma(struct drm_i915_gem_object *p, const u64 val, unsigned int count)
{
- struct page *page = __px_page(p);
- void *vaddr;
+ void *vaddr = __px_vaddr(p);
- vaddr = kmap(page);
memset64(vaddr, val, count);
clflush_cache_range(vaddr, PAGE_SIZE);
- kunmap(page);
}
static void poison_scratch_page(struct drm_i915_gem_object *scratch)
{
- struct sgt_iter sgt;
- struct page *page;
+ void *vaddr = __px_vaddr(scratch);
u8 val;
val = 0;
if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
val = POISON_FREE;
- for_each_sgt_page(page, sgt, scratch->mm.pages) {
- void *vaddr;
-
- vaddr = kmap(page);
- memset(vaddr, val, PAGE_SIZE);
- kunmap(page);
- }
+ memset(vaddr, val, scratch->base.size);
}
int setup_scratch_page(struct i915_address_space *vm)
if (IS_ERR(obj))
goto skip;
- if (pin_pt_dma(vm, obj))
+ if (map_pt_dma(vm, obj))
goto skip_obj;
/* We need a single contiguous page for our scratch */
dma_addr_t __px_dma(struct drm_i915_gem_object *p);
#define px_dma(px) (__px_dma(px_base(px)))
+void *__px_vaddr(struct drm_i915_gem_object *p);
+#define px_vaddr(px) (__px_vaddr(px_base(px)))
+
#define px_pt(px) \
__px_choose_expr(px, struct i915_page_table *, __x, \
__px_choose_expr(px, struct i915_page_directory *, &__x->pt, \
void i915_ggtt_suspend(struct i915_ggtt *gtt);
void i915_ggtt_resume(struct i915_ggtt *ggtt);
-#define kmap_atomic_px(px) kmap_atomic(__px_page(px_base(px)))
-
void
fill_page_dma(struct drm_i915_gem_object *p, const u64 val, unsigned int count);
void free_scratch(struct i915_address_space *vm);
struct drm_i915_gem_object *alloc_pt_dma(struct i915_address_space *vm, int sz);
+struct drm_i915_gem_object *alloc_pt_lmem(struct i915_address_space *vm, int sz);
struct i915_page_table *alloc_pt(struct i915_address_space *vm);
struct i915_page_directory *alloc_pd(struct i915_address_space *vm);
struct i915_page_directory *__alloc_pd(int npde);
-int pin_pt_dma(struct i915_address_space *vm, struct drm_i915_gem_object *obj);
-int pin_pt_dma_locked(struct i915_address_space *vm, struct drm_i915_gem_object *obj);
+int map_pt_dma(struct i915_address_space *vm, struct drm_i915_gem_object *obj);
+int map_pt_dma_locked(struct i915_address_space *vm, struct drm_i915_gem_object *obj);
void free_px(struct i915_address_space *vm,
struct i915_page_table *pt, int lvl);
int i915_vm_alloc_pt_stash(struct i915_address_space *vm,
struct i915_vm_pt_stash *stash,
u64 size);
-int i915_vm_pin_pt_stash(struct i915_address_space *vm,
+int i915_vm_map_pt_stash(struct i915_address_space *vm,
struct i915_vm_pt_stash *stash);
void i915_vm_free_pt_stash(struct i915_address_space *vm,
struct i915_vm_pt_stash *stash);
GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
*vaddr = i915_gem_object_pin_map(ce->state->obj,
- i915_coherent_map_type(ce->engine->i915) |
+ i915_coherent_map_type(ce->engine->i915,
+ ce->state->obj,
+ false) |
I915_MAP_OVERRIDE);
return PTR_ERR_OR_ZERO(*vaddr);
const unsigned short idx,
const u64 encoded_entry)
{
- u64 * const vaddr = kmap_atomic(__px_page(pdma));
+ u64 * const vaddr = __px_vaddr(pdma);
vaddr[idx] = encoded_entry;
clflush_cache_range(&vaddr[idx], sizeof(u64));
- kunmap_atomic(vaddr);
}
void
return 0;
}
-int i915_vm_pin_pt_stash(struct i915_address_space *vm,
+int i915_vm_map_pt_stash(struct i915_address_space *vm,
struct i915_vm_pt_stash *stash)
{
struct i915_page_table *pt;
for (n = 0; n < ARRAY_SIZE(stash->pt); n++) {
for (pt = stash->pt[n]; pt; pt = pt->stash) {
- err = pin_pt_dma_locked(vm, pt->base);
+ err = map_pt_dma_locked(vm, pt->base);
if (err)
return err;
}
return gen6_hw_domain_reset(gt, hw_mask);
}
-static int gen11_lock_sfc(struct intel_engine_cs *engine, u32 *hw_mask)
+static struct intel_engine_cs *find_sfc_paired_vecs_engine(struct intel_engine_cs *engine)
+{
+ int vecs_id;
+
+ GEM_BUG_ON(engine->class != VIDEO_DECODE_CLASS);
+
+ vecs_id = _VECS((engine->instance) / 2);
+
+ return engine->gt->engine[vecs_id];
+}
+
+struct sfc_lock_data {
+ i915_reg_t lock_reg;
+ i915_reg_t ack_reg;
+ i915_reg_t usage_reg;
+ u32 lock_bit;
+ u32 ack_bit;
+ u32 usage_bit;
+ u32 reset_bit;
+};
+
+static void get_sfc_forced_lock_data(struct intel_engine_cs *engine,
+ struct sfc_lock_data *sfc_lock)
+{
+ switch (engine->class) {
+ default:
+ MISSING_CASE(engine->class);
+ fallthrough;
+ case VIDEO_DECODE_CLASS:
+ sfc_lock->lock_reg = GEN11_VCS_SFC_FORCED_LOCK(engine);
+ sfc_lock->lock_bit = GEN11_VCS_SFC_FORCED_LOCK_BIT;
+
+ sfc_lock->ack_reg = GEN11_VCS_SFC_LOCK_STATUS(engine);
+ sfc_lock->ack_bit = GEN11_VCS_SFC_LOCK_ACK_BIT;
+
+ sfc_lock->usage_reg = GEN11_VCS_SFC_LOCK_STATUS(engine);
+ sfc_lock->usage_bit = GEN11_VCS_SFC_USAGE_BIT;
+ sfc_lock->reset_bit = GEN11_VCS_SFC_RESET_BIT(engine->instance);
+
+ break;
+ case VIDEO_ENHANCEMENT_CLASS:
+ sfc_lock->lock_reg = GEN11_VECS_SFC_FORCED_LOCK(engine);
+ sfc_lock->lock_bit = GEN11_VECS_SFC_FORCED_LOCK_BIT;
+
+ sfc_lock->ack_reg = GEN11_VECS_SFC_LOCK_ACK(engine);
+ sfc_lock->ack_bit = GEN11_VECS_SFC_LOCK_ACK_BIT;
+
+ sfc_lock->usage_reg = GEN11_VECS_SFC_USAGE(engine);
+ sfc_lock->usage_bit = GEN11_VECS_SFC_USAGE_BIT;
+ sfc_lock->reset_bit = GEN11_VECS_SFC_RESET_BIT(engine->instance);
+
+ break;
+ }
+}
+
+static int gen11_lock_sfc(struct intel_engine_cs *engine,
+ u32 *reset_mask,
+ u32 *unlock_mask)
{
struct intel_uncore *uncore = engine->uncore;
u8 vdbox_sfc_access = engine->gt->info.vdbox_sfc_access;
- i915_reg_t sfc_forced_lock, sfc_forced_lock_ack;
- u32 sfc_forced_lock_bit, sfc_forced_lock_ack_bit;
- i915_reg_t sfc_usage;
- u32 sfc_usage_bit;
- u32 sfc_reset_bit;
+ struct sfc_lock_data sfc_lock;
+ bool lock_obtained, lock_to_other = false;
int ret;
switch (engine->class) {
if ((BIT(engine->instance) & vdbox_sfc_access) == 0)
return 0;
- sfc_forced_lock = GEN11_VCS_SFC_FORCED_LOCK(engine);
- sfc_forced_lock_bit = GEN11_VCS_SFC_FORCED_LOCK_BIT;
-
- sfc_forced_lock_ack = GEN11_VCS_SFC_LOCK_STATUS(engine);
- sfc_forced_lock_ack_bit = GEN11_VCS_SFC_LOCK_ACK_BIT;
+ fallthrough;
+ case VIDEO_ENHANCEMENT_CLASS:
+ get_sfc_forced_lock_data(engine, &sfc_lock);
- sfc_usage = GEN11_VCS_SFC_LOCK_STATUS(engine);
- sfc_usage_bit = GEN11_VCS_SFC_USAGE_BIT;
- sfc_reset_bit = GEN11_VCS_SFC_RESET_BIT(engine->instance);
break;
+ default:
+ return 0;
+ }
- case VIDEO_ENHANCEMENT_CLASS:
- sfc_forced_lock = GEN11_VECS_SFC_FORCED_LOCK(engine);
- sfc_forced_lock_bit = GEN11_VECS_SFC_FORCED_LOCK_BIT;
+ if (!(intel_uncore_read_fw(uncore, sfc_lock.usage_reg) & sfc_lock.usage_bit)) {
+ struct intel_engine_cs *paired_vecs;
- sfc_forced_lock_ack = GEN11_VECS_SFC_LOCK_ACK(engine);
- sfc_forced_lock_ack_bit = GEN11_VECS_SFC_LOCK_ACK_BIT;
+ if (engine->class != VIDEO_DECODE_CLASS ||
+ !IS_GEN(engine->i915, 12))
+ return 0;
- sfc_usage = GEN11_VECS_SFC_USAGE(engine);
- sfc_usage_bit = GEN11_VECS_SFC_USAGE_BIT;
- sfc_reset_bit = GEN11_VECS_SFC_RESET_BIT(engine->instance);
- break;
+ /*
+ * Wa_14010733141
+ *
+ * If the VCS-MFX isn't using the SFC, we also need to check
+ * whether VCS-HCP is using it. If so, we need to issue a *VE*
+ * forced lock on the VE engine that shares the same SFC.
+ */
+ if (!(intel_uncore_read_fw(uncore,
+ GEN12_HCP_SFC_LOCK_STATUS(engine)) &
+ GEN12_HCP_SFC_USAGE_BIT))
+ return 0;
- default:
- return 0;
+ paired_vecs = find_sfc_paired_vecs_engine(engine);
+ get_sfc_forced_lock_data(paired_vecs, &sfc_lock);
+ lock_to_other = true;
+ *unlock_mask |= paired_vecs->mask;
+ } else {
+ *unlock_mask |= engine->mask;
}
/*
- * If the engine is using a SFC, tell the engine that a software reset
+ * If the engine is using an SFC, tell the engine that a software reset
* is going to happen. The engine will then try to force lock the SFC.
* If SFC ends up being locked to the engine we want to reset, we have
* to reset it as well (we will unlock it once the reset sequence is
* completed).
*/
- if (!(intel_uncore_read_fw(uncore, sfc_usage) & sfc_usage_bit))
- return 0;
-
- rmw_set_fw(uncore, sfc_forced_lock, sfc_forced_lock_bit);
+ rmw_set_fw(uncore, sfc_lock.lock_reg, sfc_lock.lock_bit);
ret = __intel_wait_for_register_fw(uncore,
- sfc_forced_lock_ack,
- sfc_forced_lock_ack_bit,
- sfc_forced_lock_ack_bit,
+ sfc_lock.ack_reg,
+ sfc_lock.ack_bit,
+ sfc_lock.ack_bit,
1000, 0, NULL);
- /* Was the SFC released while we were trying to lock it? */
- if (!(intel_uncore_read_fw(uncore, sfc_usage) & sfc_usage_bit))
+ /*
+ * Was the SFC released while we were trying to lock it?
+ *
+ * We should reset both the engine and the SFC if:
+ * - We were locking the SFC to this engine and the lock succeeded
+ * OR
+ * - We were locking the SFC to a different engine (Wa_14010733141)
+ * but the SFC was released before the lock was obtained.
+ *
+ * Otherwise we need only reset the engine by itself and we can
+ * leave the SFC alone.
+ */
+ lock_obtained = (intel_uncore_read_fw(uncore, sfc_lock.usage_reg) &
+ sfc_lock.usage_bit) != 0;
+ if (lock_obtained == lock_to_other)
return 0;
if (ret) {
return ret;
}
- *hw_mask |= sfc_reset_bit;
+ *reset_mask |= sfc_lock.reset_bit;
return 0;
}
{
struct intel_uncore *uncore = engine->uncore;
u8 vdbox_sfc_access = engine->gt->info.vdbox_sfc_access;
- i915_reg_t sfc_forced_lock;
- u32 sfc_forced_lock_bit;
-
- switch (engine->class) {
- case VIDEO_DECODE_CLASS:
- if ((BIT(engine->instance) & vdbox_sfc_access) == 0)
- return;
+ struct sfc_lock_data sfc_lock = {};
- sfc_forced_lock = GEN11_VCS_SFC_FORCED_LOCK(engine);
- sfc_forced_lock_bit = GEN11_VCS_SFC_FORCED_LOCK_BIT;
- break;
-
- case VIDEO_ENHANCEMENT_CLASS:
- sfc_forced_lock = GEN11_VECS_SFC_FORCED_LOCK(engine);
- sfc_forced_lock_bit = GEN11_VECS_SFC_FORCED_LOCK_BIT;
- break;
+ if (engine->class != VIDEO_DECODE_CLASS &&
+ engine->class != VIDEO_ENHANCEMENT_CLASS)
+ return;
- default:
+ if (engine->class == VIDEO_DECODE_CLASS &&
+ (BIT(engine->instance) & vdbox_sfc_access) == 0)
return;
- }
- rmw_clear_fw(uncore, sfc_forced_lock, sfc_forced_lock_bit);
+ get_sfc_forced_lock_data(engine, &sfc_lock);
+
+ rmw_clear_fw(uncore, sfc_lock.lock_reg, sfc_lock.lock_bit);
}
static int gen11_reset_engines(struct intel_gt *gt,
};
struct intel_engine_cs *engine;
intel_engine_mask_t tmp;
- u32 hw_mask;
+ u32 reset_mask, unlock_mask = 0;
int ret;
if (engine_mask == ALL_ENGINES) {
- hw_mask = GEN11_GRDOM_FULL;
+ reset_mask = GEN11_GRDOM_FULL;
} else {
- hw_mask = 0;
+ reset_mask = 0;
for_each_engine_masked(engine, gt, engine_mask, tmp) {
GEM_BUG_ON(engine->id >= ARRAY_SIZE(hw_engine_mask));
- hw_mask |= hw_engine_mask[engine->id];
- ret = gen11_lock_sfc(engine, &hw_mask);
+ reset_mask |= hw_engine_mask[engine->id];
+ ret = gen11_lock_sfc(engine, &reset_mask, &unlock_mask);
if (ret)
goto sfc_unlock;
}
}
- ret = gen6_hw_domain_reset(gt, hw_mask);
+ ret = gen6_hw_domain_reset(gt, reset_mask);
sfc_unlock:
/*
* gen11_lock_sfc to make sure that we clean properly if something
* wrong happened during the lock (e.g. lock acquired after timeout
* expiration).
+ *
+ * Due to Wa_14010733141, we may have locked an SFC to an engine that
+ * wasn't being reset. So instead of calling gen11_unlock_sfc()
+ * on engine_mask, we instead call it on the mask of engines that our
+ * gen11_lock_sfc() calls told us actually had locks attempted.
*/
- if (engine_mask != ALL_ENGINES)
- for_each_engine_masked(engine, gt, engine_mask, tmp)
- gen11_unlock_sfc(engine);
+ for_each_engine_masked(engine, gt, unlock_mask, tmp)
+ gen11_unlock_sfc(engine);
return ret;
}
int __intel_engine_reset_bh(struct intel_engine_cs *engine, const char *msg)
{
struct intel_gt *gt = engine->gt;
- bool uses_guc = intel_engine_in_guc_submission_mode(engine);
int ret;
ENGINE_TRACE(engine, "flags=%lx\n", gt->reset.flags);
"Resetting %s for %s\n", engine->name, msg);
atomic_inc(&engine->i915->gpu_error.reset_engine_count[engine->uabi_class]);
- if (!uses_guc)
- ret = intel_gt_reset_engine(engine);
- else
+ if (intel_engine_uses_guc(engine))
ret = intel_guc_reset_engine(&engine->gt->uc.guc, engine);
+ else
+ ret = intel_gt_reset_engine(engine);
if (ret) {
/* If we fail here, we expect to fallback to a global reset */
ENGINE_TRACE(engine, "Failed to reset, err: %d\n", ret);
if (unlikely(ret))
goto err_unpin;
- if (i915_vma_is_map_and_fenceable(vma))
+ if (i915_vma_is_map_and_fenceable(vma)) {
addr = (void __force *)i915_vma_pin_iomap(vma);
- else
- addr = i915_gem_object_pin_map(vma->obj,
- i915_coherent_map_type(vma->vm->i915));
+ } else {
+ int type = i915_coherent_map_type(vma->vm->i915, vma->obj, false);
+
+ addr = i915_gem_object_pin_map(vma->obj, type);
+ }
+
if (IS_ERR(addr)) {
ret = PTR_ERR(addr);
goto err_ring;
#include "intel_breadcrumbs.h"
#include "intel_context.h"
#include "intel_gt.h"
+#include "intel_gt_irq.h"
#include "intel_reset.h"
#include "intel_ring.h"
#include "shmem_utils.h"
static void i9xx_set_default_submission(struct intel_engine_cs *engine)
{
engine->submit_request = i9xx_submit_request;
-
- engine->park = NULL;
- engine->unpark = NULL;
}
static void gen6_bsd_set_default_submission(struct intel_engine_cs *engine)
{
- i9xx_set_default_submission(engine);
engine->submit_request = gen6_bsd_submit_request;
}
intel_timeline_put(engine->legacy.timeline);
}
+static void irq_handler(struct intel_engine_cs *engine, u16 iir)
+{
+ intel_engine_signal_breadcrumbs(engine);
+}
+
static void setup_irq(struct intel_engine_cs *engine)
{
struct drm_i915_private *i915 = engine->i915;
+ intel_engine_set_irq_handler(engine, irq_handler);
+
if (INTEL_GEN(i915) >= 6) {
engine->irq_enable = gen6_irq_enable;
engine->irq_disable = gen6_irq_disable;
return;
if (pm_iir & PM_VEBOX_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(gt->engine[VECS0]);
+ intel_engine_cs_irq(gt->engine[VECS0], pm_iir >> 10);
if (pm_iir & PM_VEBOX_CS_ERROR_INTERRUPT)
DRM_DEBUG("Command parser error, pm_iir 0x%08x\n", pm_iir);
return vma;
}
-__i915_active_call
static void __timeline_retire(struct i915_active *active)
{
struct intel_timeline *tl =
INIT_LIST_HEAD(&timeline->requests);
i915_syncmap_init(&timeline->sync);
- i915_active_init(&timeline->active, __timeline_active, __timeline_retire);
+ i915_active_init(&timeline->active, __timeline_active,
+ __timeline_retire, 0);
return 0;
}
wa_masked_en(wal, GEN9_ROW_CHICKEN4, GEN11_DIS_PICK_2ND_EU);
}
+/*
+ * These settings aren't actually workarounds, but general tuning settings that
+ * need to be programmed on several platforms.
+ */
+static void gen12_ctx_gt_tuning_init(struct intel_engine_cs *engine,
+ struct i915_wa_list *wal)
+{
+ /*
+ * Although some platforms refer to it as Wa_1604555607, we need to
+ * program it even on those that don't explicitly list that
+ * workaround.
+ *
+ * Note that the programming of this register is further modified
+ * according to the FF_MODE2 guidance given by Wa_1608008084:gen12.
+ * Wa_1608008084 tells us the FF_MODE2 register will return the wrong
+ * value when read. The default value for this register is zero for all
+ * fields and there are no bit masks. So instead of doing a RMW we
+ * should just write TDS timer value. For the same reason read
+ * verification is ignored.
+ */
+ wa_add(wal,
+ FF_MODE2,
+ FF_MODE2_TDS_TIMER_MASK,
+ FF_MODE2_TDS_TIMER_128,
+ 0);
+}
+
static void gen12_ctx_workarounds_init(struct intel_engine_cs *engine,
struct i915_wa_list *wal)
{
+ gen12_ctx_gt_tuning_init(engine, wal);
+
/*
* Wa_1409142259:tgl
* Wa_1409347922:tgl
wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
GEN9_PREEMPT_GPGPU_LEVEL_MASK,
GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);
-}
-
-static void tgl_ctx_workarounds_init(struct intel_engine_cs *engine,
- struct i915_wa_list *wal)
-{
- gen12_ctx_workarounds_init(engine, wal);
/*
- * Wa_1604555607:tgl,rkl
+ * Wa_16011163337
*
- * Note that the implementation of this workaround is further modified
- * according to the FF_MODE2 guidance given by Wa_1608008084:gen12.
- * FF_MODE2 register will return the wrong value when read. The default
- * value for this register is zero for all fields and there are no bit
- * masks. So instead of doing a RMW we should just write the GS Timer
- * and TDS timer values for Wa_1604555607 and Wa_16011163337.
+ * Like in gen12_ctx_gt_tuning_init(), read verification is ignored due
+ * to Wa_1608008084.
*/
wa_add(wal,
FF_MODE2,
- FF_MODE2_GS_TIMER_MASK | FF_MODE2_TDS_TIMER_MASK,
- FF_MODE2_GS_TIMER_224 | FF_MODE2_TDS_TIMER_128,
+ FF_MODE2_GS_TIMER_MASK,
+ FF_MODE2_GS_TIMER_224,
0);
}
/* Wa_22010493298 */
wa_masked_en(wal, HIZ_CHICKEN,
DG1_HZ_READ_SUPPRESSION_OPTIMIZATION_DISABLE);
-
- /*
- * Wa_16011163337
- *
- * Like in tgl_ctx_workarounds_init(), read verification is ignored due
- * to Wa_1608008084.
- */
- wa_add(wal,
- FF_MODE2,
- FF_MODE2_GS_TIMER_MASK, FF_MODE2_GS_TIMER_224, 0);
}
static void
if (IS_DG1(i915))
dg1_ctx_workarounds_init(engine, wal);
- else if (IS_ALDERLAKE_S(i915) || IS_ROCKETLAKE(i915) ||
- IS_TIGERLAKE(i915))
- tgl_ctx_workarounds_init(engine, wal);
else if (IS_GEN(i915, 12))
gen12_ctx_workarounds_init(engine, wal);
else if (IS_GEN(i915, 11))
L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS);
}
+/*
+ * Though there are per-engine instances of these registers,
+ * they retain their value through engine resets and should
+ * only be provided on the GT workaround list rather than
+ * the engine-specific workaround list.
+ */
+static void
+wa_14011060649(struct drm_i915_private *i915, struct i915_wa_list *wal)
+{
+ struct intel_engine_cs *engine;
+ struct intel_gt *gt = &i915->gt;
+ int id;
+
+ for_each_engine(engine, gt, id) {
+ if (engine->class != VIDEO_DECODE_CLASS ||
+ (engine->instance % 2))
+ continue;
+
+ wa_write_or(wal, VDBOX_CGCTL3F10(engine->mmio_base),
+ IECPUNIT_CLKGATE_DIS);
+ }
+}
+
static void
gen12_gt_workarounds_init(struct drm_i915_private *i915,
struct i915_wa_list *wal)
{
wa_init_mcr(i915, wal);
+
+ /* Wa_14011060649:tgl,rkl,dg1,adls */
+ wa_14011060649(i915, wal);
}
static void
GEN7_FF_THREAD_MODE,
GEN12_FF_TESSELATION_DOP_GATE_DISABLE);
- /* Wa_22010271021:ehl */
- if (IS_JSL_EHL(i915))
- wa_masked_en(wal,
- GEN9_CS_DEBUG_MODE1,
- FF_DOP_CLOCK_GATE_DISABLE);
+ /* Wa_22010271021 */
+ wa_masked_en(wal,
+ GEN9_CS_DEBUG_MODE1,
+ FF_DOP_CLOCK_GATE_DISABLE);
}
if (IS_GEN_RANGE(i915, 9, 12)) {
CACHE_MODE_0_GEN7,
/* enable HiZ Raw Stall Optimization */
HIZ_RAW_STALL_OPT_DISABLE);
-
- /* WaDisable4x2SubspanOptimization:hsw */
- wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
}
if (IS_VALLEYVIEW(i915)) {
kfree(ring);
return NULL;
}
- i915_active_init(&ring->vma->active, NULL, NULL);
+ i915_active_init(&ring->vma->active, NULL, NULL, 0);
__set_bit(I915_VMA_GGTT_BIT, __i915_vma_flags(ring->vma));
__set_bit(DRM_MM_NODE_ALLOCATED_BIT, &ring->vma->node.flags);
ring->vma->node.size = sz;
goto err;
vaddr = i915_gem_object_pin_map_unlocked(ce->state->obj,
- i915_coherent_map_type(engine->i915));
+ i915_coherent_map_type(engine->i915,
+ ce->state->obj, false));
if (IS_ERR(vaddr)) {
err = PTR_ERR(vaddr);
intel_context_unpin(ce);
return p;
kref_init(&p->kref);
- i915_active_init(&p->active, pulse_active, pulse_retire);
+ i915_active_init(&p->active, pulse_active, pulse_retire, 0);
return p;
}
SUBTEST(live_virtual_reset),
};
- if (!HAS_EXECLISTS(i915))
+ if (i915->gt.submission_method != INTEL_SUBMISSION_ELSP)
return 0;
if (intel_gt_is_wedged(&i915->gt))
h->seqno = memset(vaddr, 0xff, PAGE_SIZE);
vaddr = i915_gem_object_pin_map_unlocked(h->obj,
- i915_coherent_map_type(gt->i915));
+ i915_coherent_map_type(gt->i915, h->obj, false));
if (IS_ERR(vaddr)) {
err = PTR_ERR(vaddr);
goto err_unpin_hws;
return ERR_CAST(obj);
}
- vaddr = i915_gem_object_pin_map_unlocked(obj, i915_coherent_map_type(gt->i915));
+ vaddr = i915_gem_object_pin_map_unlocked(obj, i915_coherent_map_type(gt->i915, obj, false));
if (IS_ERR(vaddr)) {
i915_gem_object_put(obj);
i915_vm_put(vm);
}
lrc = i915_gem_object_pin_map_unlocked(ce->state->obj,
- i915_coherent_map_type(engine->i915));
+ i915_coherent_map_type(engine->i915,
+ ce->state->obj,
+ false));
if (IS_ERR(lrc)) {
err = PTR_ERR(lrc);
goto err_B1;
struct intel_rc6 *rc6 = >->rc6;
u64 rc0_power, rc6_power;
intel_wakeref_t wakeref;
+ bool has_power;
ktime_t dt;
u64 res[2];
int err = 0;
if (IS_VALLEYVIEW(gt->i915) || IS_CHERRYVIEW(gt->i915))
return 0;
+ has_power = librapl_supported(gt->i915);
wakeref = intel_runtime_pm_get(gt->uncore->rpm);
/* Force RC6 off for starters */
goto out_unlock;
}
- rc0_power = div64_u64(NSEC_PER_SEC * rc0_power, ktime_to_ns(dt));
- if (!rc0_power) {
- pr_err("No power measured while in RC0\n");
- err = -EINVAL;
- goto out_unlock;
+ if (has_power) {
+ rc0_power = div64_u64(NSEC_PER_SEC * rc0_power,
+ ktime_to_ns(dt));
+ if (!rc0_power) {
+ pr_err("No power measured while in RC0\n");
+ err = -EINVAL;
+ goto out_unlock;
+ }
}
/* Manually enter RC6 */
err = -EINVAL;
}
- rc6_power = div64_u64(NSEC_PER_SEC * rc6_power, ktime_to_ns(dt));
- pr_info("GPU consumed %llduW in RC0 and %llduW in RC6\n",
- rc0_power, rc6_power);
- if (2 * rc6_power > rc0_power) {
- pr_err("GPU leaked energy while in RC6!\n");
- err = -EINVAL;
- goto out_unlock;
+ if (has_power) {
+ rc6_power = div64_u64(NSEC_PER_SEC * rc6_power,
+ ktime_to_ns(dt));
+ pr_info("GPU consumed %llduW in RC0 and %llduW in RC6\n",
+ rc0_power, rc6_power);
+ if (2 * rc6_power > rc0_power) {
+ pr_err("GPU leaked energy while in RC6!\n");
+ err = -EINVAL;
+ goto out_unlock;
+ }
}
/* Restore what should have been the original state! */
SUBTEST(live_ctx_switch_wa),
};
- if (HAS_EXECLISTS(i915))
+ if (i915->gt.submission_method > INTEL_SUBMISSION_RING)
return 0;
return intel_gt_live_subtests(tests, &i915->gt);
int err = 0;
/*
- * The premise is that the GPU does change freqency at our behest.
+ * The premise is that the GPU does change frequency at our behest.
* Let's check there is a correspondence between the requested
* frequency, the actual frequency, and the observed clock rate.
*/
int err = 0;
/*
- * The premise is that the GPU does change freqency at our behest.
+ * The premise is that the GPU does change frequency at our behest.
* Let's check there is a correspondence between the requested
* frequency, the actual frequency, and the observed clock rate.
*/
if (!intel_rps_is_enabled(rps) || INTEL_GEN(gt->i915) < 6)
return 0;
- if (!librapl_energy_uJ())
+ if (!librapl_supported(gt->i915))
return 0;
if (igt_spinner_init(&spin, gt))
#include <linux/shmem_fs.h>
#include "gem/i915_gem_object.h"
+#include "gem/i915_gem_lmem.h"
#include "shmem_utils.h"
struct file *shmem_create_from_data(const char *name, void *data, size_t len)
return file;
}
- ptr = i915_gem_object_pin_map_unlocked(obj, I915_MAP_WB);
+ ptr = i915_gem_object_pin_map_unlocked(obj, i915_gem_object_is_lmem(obj) ?
+ I915_MAP_WC : I915_MAP_WB);
if (IS_ERR(ptr))
return ERR_CAST(ptr);
if (IS_ERR(vma))
return PTR_ERR(vma);
- vaddr = i915_gem_object_pin_map_unlocked(vma->obj, I915_MAP_WB);
+ vaddr = i915_gem_object_pin_map_unlocked(vma->obj,
+ i915_coherent_map_type(guc_to_gt(guc)->i915,
+ vma->obj, true));
if (IS_ERR(vaddr)) {
i915_vma_unpin_and_release(&vma, 0);
return PTR_ERR(vaddr);
#include "gt/intel_context.h"
#include "gt/intel_engine_pm.h"
#include "gt/intel_gt.h"
+#include "gt/intel_gt_irq.h"
#include "gt/intel_gt_pm.h"
#include "gt/intel_lrc.h"
#include "gt/intel_mocs.h"
spin_unlock_irqrestore(&engine->active.lock, flags);
}
+static void cs_irq_handler(struct intel_engine_cs *engine, u16 iir)
+{
+ if (iir & GT_RENDER_USER_INTERRUPT) {
+ intel_engine_signal_breadcrumbs(engine);
+ tasklet_hi_schedule(&engine->execlists.tasklet);
+ }
+}
+
static void guc_reset_prepare(struct intel_engine_cs *engine)
{
struct intel_engine_execlists * const execlists = &engine->execlists;
static void guc_set_default_submission(struct intel_engine_cs *engine)
{
engine->submit_request = guc_submit_request;
- engine->schedule = i915_schedule;
- engine->execlists.tasklet.callback = guc_submission_tasklet;
-
- engine->reset.prepare = guc_reset_prepare;
- engine->reset.rewind = guc_reset_rewind;
- engine->reset.cancel = guc_reset_cancel;
- engine->reset.finish = guc_reset_finish;
-
- engine->flags |= I915_ENGINE_NEEDS_BREADCRUMB_TASKLET;
- engine->flags |= I915_ENGINE_HAS_PREEMPTION;
-
- /*
- * TODO: GuC supports timeslicing and semaphores as well, but they're
- * handled by the firmware so some minor tweaks are required before
- * enabling.
- *
- * engine->flags |= I915_ENGINE_HAS_TIMESLICES;
- * engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
- */
-
- engine->emit_bb_start = gen8_emit_bb_start;
-
- /*
- * For the breadcrumb irq to work we need the interrupts to stay
- * enabled. However, on all platforms on which we'll have support for
- * GuC submission we don't allow disabling the interrupts at runtime, so
- * we're always safe with the current flow.
- */
- GEM_BUG_ON(engine->irq_enable || engine->irq_disable);
}
static void guc_release(struct intel_engine_cs *engine)
engine->cops = &guc_context_ops;
engine->request_alloc = guc_request_alloc;
+ engine->schedule = i915_schedule;
+
+ engine->reset.prepare = guc_reset_prepare;
+ engine->reset.rewind = guc_reset_rewind;
+ engine->reset.cancel = guc_reset_cancel;
+ engine->reset.finish = guc_reset_finish;
+
engine->emit_flush = gen8_emit_flush_xcs;
engine->emit_init_breadcrumb = gen8_emit_init_breadcrumb;
engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb_xcs;
engine->emit_flush = gen12_emit_flush_xcs;
}
engine->set_default_submission = guc_set_default_submission;
+
+ engine->flags |= I915_ENGINE_HAS_PREEMPTION;
+
+ /*
+ * TODO: GuC supports timeslicing and semaphores as well, but they're
+ * handled by the firmware so some minor tweaks are required before
+ * enabling.
+ *
+ * engine->flags |= I915_ENGINE_HAS_TIMESLICES;
+ * engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
+ */
+
+ engine->emit_bb_start = gen8_emit_bb_start;
}
static void rcs_submission_override(struct intel_engine_cs *engine)
static inline void guc_default_irqs(struct intel_engine_cs *engine)
{
engine->irq_keep_mask = GT_RENDER_USER_INTERRUPT;
+ intel_engine_set_irq_handler(engine, cs_irq_handler);
}
int intel_guc_submission_setup(struct intel_engine_cs *engine)
{
guc->submission_selected = __guc_submission_selected(guc);
}
-
-bool intel_engine_in_guc_submission_mode(const struct intel_engine_cs *engine)
-{
- return engine->set_default_submission == guc_set_default_submission;
-}
int intel_guc_preempt_work_create(struct intel_guc *guc);
void intel_guc_preempt_work_destroy(struct intel_guc *guc);
int intel_guc_submission_setup(struct intel_engine_cs *engine);
-bool intel_engine_in_guc_submission_mode(const struct intel_engine_cs *engine);
static inline bool intel_guc_submission_is_supported(struct intel_guc *guc)
{
if (IS_ERR(vma))
return PTR_ERR(vma);
- vaddr = i915_gem_object_pin_map_unlocked(vma->obj, I915_MAP_WB);
+ vaddr = i915_gem_object_pin_map_unlocked(vma->obj,
+ i915_coherent_map_type(gt->i915,
+ vma->obj, true));
if (IS_ERR(vaddr)) {
i915_vma_unpin_and_release(&vma, 0);
return PTR_ERR(vaddr);
void __i915_active_init(struct i915_active *ref,
int (*active)(struct i915_active *ref),
void (*retire)(struct i915_active *ref),
+ unsigned long flags,
struct lock_class_key *mkey,
struct lock_class_key *wkey)
{
- unsigned long bits;
-
debug_active_init(ref);
- ref->flags = 0;
+ ref->flags = flags;
ref->active = active;
- ref->retire = ptr_unpack_bits(retire, &bits, 2);
- if (bits & I915_ACTIVE_MAY_SLEEP)
- ref->flags |= I915_ACTIVE_RETIRE_SLEEPS;
+ ref->retire = retire;
spin_lock_init(&ref->tree_lock);
ref->tree = RB_ROOT;
return 0;
}
-__i915_active_call static void
-auto_retire(struct i915_active *ref)
+static void auto_retire(struct i915_active *ref)
{
i915_active_put(ref);
}
return NULL;
kref_init(&aa->ref);
- i915_active_init(&aa->base, auto_active, auto_retire);
+ i915_active_init(&aa->base, auto_active, auto_retire, 0);
return &aa->base;
}
void __i915_active_init(struct i915_active *ref,
int (*active)(struct i915_active *ref),
void (*retire)(struct i915_active *ref),
+ unsigned long flags,
struct lock_class_key *mkey,
struct lock_class_key *wkey);
/* Specialise each class of i915_active to avoid impossible lockdep cycles. */
-#define i915_active_init(ref, active, retire) do { \
- static struct lock_class_key __mkey; \
- static struct lock_class_key __wkey; \
- \
- __i915_active_init(ref, active, retire, &__mkey, &__wkey); \
+#define i915_active_init(ref, active, retire, flags) do { \
+ static struct lock_class_key __mkey; \
+ static struct lock_class_key __wkey; \
+ \
+ __i915_active_init(ref, active, retire, flags, &__mkey, &__wkey); \
} while (0)
struct dma_fence *
struct active_node;
-#define I915_ACTIVE_MAY_SLEEP BIT(0)
-
-#define __i915_active_call __aligned(4)
-#define i915_active_may_sleep(fn) ptr_pack_bits(&(fn), I915_ACTIVE_MAY_SLEEP, 2)
-
struct i915_active {
atomic_t count;
struct mutex mutex;
return 0;
}
+/**
+ * intel_engine_cmd_parser_alloc_jump_whitelist() - preallocate jump whitelist for intel_engine_cmd_parser()
+ * @batch_length: length of the commands in batch_obj
+ * @trampoline: Whether jump trampolines are used.
+ *
+ * Preallocates a jump whitelist for parsing the cmd buffer in intel_engine_cmd_parser().
+ * This has to be preallocated, because the command parser runs in signaling context,
+ * and may not allocate any memory.
+ *
+ * Return: NULL or pointer to a jump whitelist, or ERR_PTR() on failure. Use
+ * IS_ERR() to check for errors. Must bre freed() with kfree().
+ *
+ * NULL is a valid value, meaning no allocation was required.
+ */
unsigned long *intel_engine_cmd_parser_alloc_jump_whitelist(u32 batch_length,
bool trampoline)
{
* @batch_offset: byte offset in the batch at which execution starts
* @batch_length: length of the commands in batch_obj
* @shadow: validated copy of the batch buffer in question
- * @trampoline: whether to emit a conditional trampoline at the end of the batch
+ * @jump_whitelist: buffer preallocated with intel_engine_cmd_parser_alloc_jump_whitelist()
+ * @shadow_map: mapping to @shadow vma
+ * @batch_map: mapping to @batch vma
*
* Parses the specified batch buffer looking for privilege violations as
* described in the overview.
seq_printf(m, "DDC2 = 0x%08x\n",
intel_uncore_read(uncore, DCC2));
seq_printf(m, "C0DRB3 = 0x%04x\n",
- intel_uncore_read16(uncore, C0DRB3));
+ intel_uncore_read16(uncore, C0DRB3_BW));
seq_printf(m, "C1DRB3 = 0x%04x\n",
- intel_uncore_read16(uncore, C1DRB3));
+ intel_uncore_read16(uncore, C1DRB3_BW));
} else if (INTEL_GEN(dev_priv) >= 6) {
seq_printf(m, "MAD_DIMM_C0 = 0x%08x\n",
intel_uncore_read(uncore, MAD_DIMM_C0));
DRM_IOCTL_DEF_DRV(I915_GEM_ENTERVT, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
DRM_IOCTL_DEF_DRV(I915_GEM_LEAVEVT, drm_noop, DRM_AUTH|DRM_MASTER|DRM_ROOT_ONLY),
DRM_IOCTL_DEF_DRV(I915_GEM_CREATE, i915_gem_create_ioctl, DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF_DRV(I915_GEM_CREATE_EXT, i915_gem_create_ext_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(I915_GEM_PREAD, i915_gem_pread_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(I915_GEM_PWRITE, i915_gem_pwrite_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(I915_GEM_MMAP, i915_gem_mmap_ioctl, DRM_RENDER_ALLOW),
#include "gem/i915_gem_context_types.h"
#include "gem/i915_gem_shrinker.h"
#include "gem/i915_gem_stolen.h"
+#include "gem/i915_gem_lmem.h"
#include "gt/intel_engine.h"
#include "gt/intel_gt_types.h"
};
struct i915_gem_mm {
+ /*
+ * Shortcut for the stolen region. This points to either
+ * INTEL_REGION_STOLEN_SMEM for integrated platforms, or
+ * INTEL_REGION_STOLEN_LMEM for discrete, or NULL if the device doesn't
+ * support stolen.
+ */
+ struct intel_memory_region *stolen_region;
/** Memory allocator for GTT stolen memory */
struct drm_mm stolen;
/** Protects the usage of the GTT stolen memory allocator. This is
}
static inline bool
-intel_ggtt_update_needs_vtd_wa(struct drm_i915_private *dev_priv)
+intel_ggtt_update_needs_vtd_wa(struct drm_i915_private *i915)
{
- return IS_BROXTON(dev_priv) && intel_vtd_active();
+ return IS_BROXTON(i915) && intel_vtd_active();
+}
+
+static inline bool
+intel_vm_no_concurrent_access_wa(struct drm_i915_private *i915)
+{
+ return IS_CHERRYVIEW(i915) || intel_ggtt_update_needs_vtd_wa(i915);
}
/* i915_drv.c */
#define I915_GEM_OBJECT_UNBIND_ACTIVE BIT(0)
#define I915_GEM_OBJECT_UNBIND_BARRIER BIT(1)
#define I915_GEM_OBJECT_UNBIND_TEST BIT(2)
+#define I915_GEM_OBJECT_UNBIND_VM_TRYLOCK BIT(3)
void i915_gem_runtime_suspend(struct drm_i915_private *dev_priv);
}
static inline enum i915_map_type
-i915_coherent_map_type(struct drm_i915_private *i915)
+i915_coherent_map_type(struct drm_i915_private *i915,
+ struct drm_i915_gem_object *obj, bool always_coherent)
{
- return HAS_LLC(i915) ? I915_MAP_WB : I915_MAP_WC;
+ if (i915_gem_object_is_lmem(obj))
+ return I915_MAP_WC;
+ if (HAS_LLC(i915) || always_coherent)
+ return I915_MAP_WB;
+ else
+ return I915_MAP_WC;
}
#endif
if (vma) {
ret = -EBUSY;
if (flags & I915_GEM_OBJECT_UNBIND_ACTIVE ||
- !i915_vma_is_active(vma))
- ret = i915_vma_unbind(vma);
+ !i915_vma_is_active(vma)) {
+ if (flags & I915_GEM_OBJECT_UNBIND_VM_TRYLOCK) {
+ if (mutex_trylock(&vma->vm->mutex)) {
+ ret = __i915_vma_unbind(vma);
+ mutex_unlock(&vma->vm->mutex);
+ } else {
+ ret = -EBUSY;
+ }
+ } else {
+ ret = i915_vma_unbind(vma);
+ }
+ }
__i915_vma_put(vma);
}
intel_uncore_write16(&dev_priv->uncore, GEN2_IIR, iir);
if (iir & I915_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(dev_priv->gt.engine[RCS0]);
+ intel_engine_cs_irq(dev_priv->gt.engine[RCS0], iir);
if (iir & I915_MASTER_ERROR_INTERRUPT)
i8xx_error_irq_handler(dev_priv, eir, eir_stuck);
intel_uncore_write(&dev_priv->uncore, GEN2_IIR, iir);
if (iir & I915_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(dev_priv->gt.engine[RCS0]);
+ intel_engine_cs_irq(dev_priv->gt.engine[RCS0], iir);
if (iir & I915_MASTER_ERROR_INTERRUPT)
i9xx_error_irq_handler(dev_priv, eir, eir_stuck);
intel_uncore_write(&dev_priv->uncore, GEN2_IIR, iir);
if (iir & I915_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(dev_priv->gt.engine[RCS0]);
+ intel_engine_cs_irq(dev_priv->gt.engine[RCS0],
+ iir);
if (iir & I915_BSD_USER_INTERRUPT)
- intel_engine_signal_breadcrumbs(dev_priv->gt.engine[VCS0]);
+ intel_engine_cs_irq(dev_priv->gt.engine[VCS0],
+ iir >> 25);
if (iir & I915_MASTER_ERROR_INTERRUPT)
i9xx_error_irq_handler(dev_priv, eir, eir_stuck);
param(int, fastboot, -1, 0600) \
param(int, enable_dpcd_backlight, -1, 0600) \
param(char *, force_probe, CONFIG_DRM_I915_FORCE_PROBE, 0400) \
- param(unsigned long, fake_lmem_start, 0, 0400) \
- param(unsigned int, request_timeout_ms, CONFIG_DRM_I915_REQUEST_TIMEOUT, 0600) \
+ param(unsigned long, fake_lmem_start, 0, IS_ENABLED(CONFIG_DRM_I915_UNSTABLE_FAKE_LMEM) ? 0400 : 0) \
+ param(unsigned int, request_timeout_ms, CONFIG_DRM_I915_REQUEST_TIMEOUT, CONFIG_DRM_I915_REQUEST_TIMEOUT ? 0600 : 0) \
/* leave bools at the end to not create holes */ \
param(bool, enable_hangcheck, true, 0600) \
param(bool, load_detect_test, false, 0600) \
param(bool, force_reset_modeset_test, false, 0600) \
- param(bool, error_capture, true, 0600) \
+ param(bool, error_capture, true, IS_ENABLED(CONFIG_DRM_I915_CAPTURE_ERROR) ? 0600 : 0) \
param(bool, disable_display, false, 0400) \
param(bool, verbose_state_checks, true, 0) \
param(bool, nuclear_pageflip, false, 0400) \
param(bool, enable_dp_mst, true, 0600) \
- param(bool, enable_gvt, false, 0400)
+ param(bool, enable_gvt, false, IS_ENABLED(CONFIG_DRM_I915_GVT) ? 0400 : 0)
#define MEMBER(T, member, ...) T member;
struct i915_params {
};
#define DGFX_FEATURES \
- .memory_regions = REGION_SMEM | REGION_LMEM, \
+ .memory_regions = REGION_SMEM | REGION_LMEM | REGION_STOLEN_LMEM, \
.has_master_unit_irq = 1, \
.has_llc = 0, \
.has_snoop = 1, \
case 8:
case 9:
case 10:
- if (intel_engine_in_execlists_submission_mode(ce->engine)) {
- stream->specific_ctx_id_mask =
- (1U << GEN8_CTX_ID_WIDTH) - 1;
- stream->specific_ctx_id = stream->specific_ctx_id_mask;
- } else {
+ if (intel_engine_uses_guc(ce->engine)) {
/*
* When using GuC, the context descriptor we write in
* i915 is read by GuC and rewritten before it's
*/
stream->specific_ctx_id_mask =
(1U << (GEN8_CTX_ID_WIDTH - 1)) - 1;
+ } else {
+ stream->specific_ctx_id_mask =
+ (1U << GEN8_CTX_ID_WIDTH) - 1;
+ stream->specific_ctx_id = stream->specific_ctx_id_mask;
}
break;
static int
config_status(struct drm_i915_private *i915, u64 config)
{
+ struct intel_gt *gt = &i915->gt;
+
switch (config) {
case I915_PMU_ACTUAL_FREQUENCY:
if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))
case I915_PMU_INTERRUPTS:
break;
case I915_PMU_RC6_RESIDENCY:
- if (!HAS_RC6(i915))
+ if (!gt->rc6.supported)
return -ENODEV;
break;
case I915_PMU_SOFTWARE_GT_AWAKE_TIME:
}
}
+static int query_memregion_info(struct drm_i915_private *i915,
+ struct drm_i915_query_item *query_item)
+{
+ struct drm_i915_query_memory_regions __user *query_ptr =
+ u64_to_user_ptr(query_item->data_ptr);
+ struct drm_i915_memory_region_info __user *info_ptr =
+ &query_ptr->regions[0];
+ struct drm_i915_memory_region_info info = { };
+ struct drm_i915_query_memory_regions query;
+ struct intel_memory_region *mr;
+ u32 total_length;
+ int ret, id, i;
+
+ if (!IS_ENABLED(CONFIG_DRM_I915_UNSTABLE_FAKE_LMEM))
+ return -ENODEV;
+
+ if (query_item->flags != 0)
+ return -EINVAL;
+
+ total_length = sizeof(query);
+ for_each_memory_region(mr, i915, id) {
+ if (mr->private)
+ continue;
+
+ total_length += sizeof(info);
+ }
+
+ ret = copy_query_item(&query, sizeof(query), total_length, query_item);
+ if (ret != 0)
+ return ret;
+
+ if (query.num_regions)
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(query.rsvd); i++) {
+ if (query.rsvd[i])
+ return -EINVAL;
+ }
+
+ for_each_memory_region(mr, i915, id) {
+ if (mr->private)
+ continue;
+
+ info.region.memory_class = mr->type;
+ info.region.memory_instance = mr->instance;
+ info.probed_size = mr->total;
+ info.unallocated_size = mr->avail;
+
+ if (__copy_to_user(info_ptr, &info, sizeof(info)))
+ return -EFAULT;
+
+ query.num_regions++;
+ info_ptr++;
+ }
+
+ if (__copy_to_user(query_ptr, &query, sizeof(query)))
+ return -EFAULT;
+
+ return total_length;
+}
+
static int (* const i915_query_funcs[])(struct drm_i915_private *dev_priv,
struct drm_i915_query_item *query_item) = {
query_topology_info,
query_engine_info,
query_perf_config,
+ query_memregion_info,
};
int i915_query_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
#define GEN11_VECS_SFC_USAGE(engine) _MMIO((engine)->mmio_base + 0x2014)
#define GEN11_VECS_SFC_USAGE_BIT (1 << 0)
+#define GEN12_HCP_SFC_FORCED_LOCK(engine) _MMIO((engine)->mmio_base + 0x2910)
+#define GEN12_HCP_SFC_FORCED_LOCK_BIT REG_BIT(0)
+#define GEN12_HCP_SFC_LOCK_STATUS(engine) _MMIO((engine)->mmio_base + 0x2914)
+#define GEN12_HCP_SFC_LOCK_ACK_BIT REG_BIT(1)
+#define GEN12_HCP_SFC_USAGE_BIT REG_BIT(0)
+
#define GEN12_SFC_DONE(n) _MMIO(0x1cc00 + (n) * 0x100)
#define GEN12_SFC_DONE_MAX 4
#define GAB_CTL _MMIO(0x24000)
#define GAB_CTL_CONT_AFTER_PAGEFAULT (1 << 8)
+#define GU_CNTL _MMIO(0x101010)
+#define LMEM_INIT REG_BIT(7)
+
#define GEN6_STOLEN_RESERVED _MMIO(0x1082C0)
#define GEN6_STOLEN_RESERVED_ADDR_MASK (0xFFF << 20)
#define GEN7_STOLEN_RESERVED_ADDR_MASK (0x3FFF << 18)
#define RING_INDIRECT_CTX_OFFSET(base) _MMIO((base) + 0x1c8) /* gen8+ */
#define RING_CTX_TIMESTAMP(base) _MMIO((base) + 0x3a8) /* gen8+ */
+#define VDBOX_CGCTL3F10(base) _MMIO((base) + 0x3f10)
+#define IECPUNIT_CLKGATE_DIS REG_BIT(22)
+
#define ERROR_GEN6 _MMIO(0x40a0)
#define GEN7_ERR_INT _MMIO(0x44040)
#define ERR_INT_POISON (1 << 31)
#define CSHRDDR3CTL_DDR3 (1 << 2)
/* 965 MCH register controlling DRAM channel configuration */
-#define C0DRB3 _MMIO(MCHBAR_MIRROR_BASE + 0x206)
-#define C1DRB3 _MMIO(MCHBAR_MIRROR_BASE + 0x606)
+#define C0DRB3_BW _MMIO(MCHBAR_MIRROR_BASE + 0x206)
+#define C1DRB3_BW _MMIO(MCHBAR_MIRROR_BASE + 0x606)
/* snb MCH registers for reading the DRAM channel configuration */
#define MAD_DIMM_C0 _MMIO(MCHBAR_MIRROR_BASE_SNB + 0x5004)
#define GEN12_GLOBAL_MOCS(i) _MMIO(0x4000 + (i) * 4) /* Global MOCS regs */
#define GEN12_GSMBASE _MMIO(0x108100)
+#define GEN12_DSMBASE _MMIO(0x1080C0)
/* gamt regs */
#define GEN8_L3_LRA_1_GPGPU _MMIO(0x4dd4)
u32 seqno;
int ret;
- might_sleep_if(gfpflags_allow_blocking(gfp));
+ might_alloc(gfp);
/* Check that the caller provided an already pinned context */
__intel_context_pin(ce);
#include "display/intel_frontbuffer.h"
+#include "gem/i915_gem_lmem.h"
#include "gt/intel_engine.h"
#include "gt/intel_engine_heartbeat.h"
#include "gt/intel_gt.h"
return i915_vma_tryget(active_to_vma(ref)) ? 0 : -ENOENT;
}
-__i915_active_call
static void __i915_vma_retire(struct i915_active *ref)
{
i915_vma_put(active_to_vma(ref));
vma->size = obj->base.size;
vma->display_alignment = I915_GTT_MIN_ALIGNMENT;
- i915_active_init(&vma->active, __i915_vma_active, __i915_vma_retire);
+ i915_active_init(&vma->active, __i915_vma_active, __i915_vma_retire, 0);
/* Declare ourselves safe for use inside shrinkers */
if (IS_ENABLED(CONFIG_LOCKDEP)) {
void __iomem *ptr;
int err;
- if (GEM_WARN_ON(!i915_vma_is_map_and_fenceable(vma))) {
- err = -ENODEV;
- goto err;
+ if (!i915_gem_object_is_lmem(vma->obj)) {
+ if (GEM_WARN_ON(!i915_vma_is_map_and_fenceable(vma))) {
+ err = -ENODEV;
+ goto err;
+ }
}
GEM_BUG_ON(!i915_vma_is_ggtt(vma));
ptr = READ_ONCE(vma->iomap);
if (ptr == NULL) {
- ptr = io_mapping_map_wc(&i915_vm_to_ggtt(vma->vm)->iomap,
- vma->node.start,
- vma->node.size);
+ /*
+ * TODO: consider just using i915_gem_object_pin_map() for lmem
+ * instead, which already supports mapping non-contiguous chunks
+ * of pages, that way we can also drop the
+ * I915_BO_ALLOC_CONTIGUOUS when allocating the object.
+ */
+ if (i915_gem_object_is_lmem(vma->obj))
+ ptr = i915_gem_object_lmem_io_map(vma->obj, 0,
+ vma->obj->base.size);
+ else
+ ptr = io_mapping_map_wc(&i915_vm_to_ggtt(vma->vm)->iomap,
+ vma->node.start,
+ vma->node.size);
if (ptr == NULL) {
err = -ENOMEM;
goto err;
if (err)
goto err_fence;
- err = i915_vm_pin_pt_stash(vma->vm,
- &work->stash);
+ err = i915_vm_map_pt_stash(vma->vm, &work->stash);
if (err)
goto err_fence;
}
.class = INTEL_MEMORY_STOLEN_SYSTEM,
.instance = 0,
},
+ [INTEL_REGION_STOLEN_LMEM] = {
+ .class = INTEL_MEMORY_STOLEN_LOCAL,
+ .instance = 0,
+ },
};
+struct intel_memory_region *
+intel_memory_region_lookup(struct drm_i915_private *i915,
+ u16 class, u16 instance)
+{
+ struct intel_memory_region *mr;
+ int id;
+
+ /* XXX: consider maybe converting to an rb tree at some point */
+ for_each_memory_region(mr, i915, id) {
+ if (mr->type == class && mr->instance == instance)
+ return mr;
+ }
+
+ return NULL;
+}
+
struct intel_memory_region *
intel_memory_region_by_type(struct drm_i915_private *i915,
enum intel_memory_type mem_type)
case INTEL_MEMORY_SYSTEM:
mem = i915_gem_shmem_setup(i915);
break;
+ case INTEL_MEMORY_STOLEN_LOCAL:
+ mem = i915_gem_stolen_lmem_setup(i915);
+ if (!IS_ERR(mem))
+ i915->mm.stolen_region = mem;
+ break;
case INTEL_MEMORY_STOLEN_SYSTEM:
- mem = i915_gem_stolen_setup(i915);
+ mem = i915_gem_stolen_smem_setup(i915);
+ if (!IS_ERR(mem))
+ i915->mm.stolen_region = mem;
break;
default:
continue;
#include <linux/mutex.h>
#include <linux/io-mapping.h>
#include <drm/drm_mm.h>
+#include <drm/i915_drm.h>
#include "i915_buddy.h"
struct intel_memory_region;
struct sg_table;
-/**
- * Base memory type
- */
enum intel_memory_type {
- INTEL_MEMORY_SYSTEM = 0,
- INTEL_MEMORY_LOCAL,
+ INTEL_MEMORY_SYSTEM = I915_MEMORY_CLASS_SYSTEM,
+ INTEL_MEMORY_LOCAL = I915_MEMORY_CLASS_DEVICE,
INTEL_MEMORY_STOLEN_SYSTEM,
+ INTEL_MEMORY_STOLEN_LOCAL,
};
enum intel_region_id {
INTEL_REGION_SMEM = 0,
INTEL_REGION_LMEM,
INTEL_REGION_STOLEN_SMEM,
+ INTEL_REGION_STOLEN_LMEM,
INTEL_REGION_UNKNOWN, /* Should be last */
};
#define REGION_SMEM BIT(INTEL_REGION_SMEM)
#define REGION_LMEM BIT(INTEL_REGION_LMEM)
#define REGION_STOLEN_SMEM BIT(INTEL_REGION_STOLEN_SMEM)
+#define REGION_STOLEN_LMEM BIT(INTEL_REGION_STOLEN_LMEM)
#define I915_ALLOC_MIN_PAGE_SIZE BIT(0)
#define I915_ALLOC_CONTIGUOUS BIT(1)
u16 type;
u16 instance;
enum intel_region_id id;
- char name[8];
+ char name[16];
+ bool private; /* not for userspace */
struct list_head reserved;
} objects;
};
+struct intel_memory_region *
+intel_memory_region_lookup(struct drm_i915_private *i915,
+ u16 class, u16 instance);
+
int intel_memory_region_init_buddy(struct intel_memory_region *mem);
void intel_memory_region_release_buddy(struct intel_memory_region *mem);
if (ret)
return ret;
+ /*
+ * The boot firmware initializes local memory and assesses its health.
+ * If memory training fails, the punit will have been instructed to
+ * keep the GT powered down; we won't be able to communicate with it
+ * and we should not continue with driver initialization.
+ */
+ if (IS_DGFX(i915) &&
+ !(__raw_uncore_read32(uncore, GU_CNTL) & LMEM_INIT)) {
+ drm_err(&i915->drm, "LMEM not initialized by firmware\n");
+ return -ENODEV;
+ }
+
if (INTEL_GEN(i915) > 5 && !intel_vgpu_active(i915))
uncore->flags |= UNCORE_HAS_FORCEWAKE;
return NULL;
kref_init(&active->ref);
- i915_active_init(&active->base, __live_active, __live_retire);
+ i915_active_init(&active->base, __live_active, __live_retire, 0);
return active;
}
intel_runtime_pm_put(&i915->runtime_pm, wakeref);
}
-static int pm_prepare(struct drm_i915_private *i915)
+static int igt_pm_prepare(struct drm_i915_private *i915)
{
i915_gem_suspend(i915);
return 0;
}
-static void pm_suspend(struct drm_i915_private *i915)
+static void igt_pm_suspend(struct drm_i915_private *i915)
{
intel_wakeref_t wakeref;
}
}
-static void pm_hibernate(struct drm_i915_private *i915)
+static void igt_pm_hibernate(struct drm_i915_private *i915)
{
intel_wakeref_t wakeref;
}
}
-static void pm_resume(struct drm_i915_private *i915)
+static void igt_pm_resume(struct drm_i915_private *i915)
{
intel_wakeref_t wakeref;
if (err)
goto out;
- err = pm_prepare(i915);
+ err = igt_pm_prepare(i915);
if (err)
goto out;
- pm_suspend(i915);
+ igt_pm_suspend(i915);
/* Here be dragons! Note that with S3RST any S3 may become S4! */
simulate_hibernate(i915);
- pm_resume(i915);
+ igt_pm_resume(i915);
err = switch_to_context(ctx);
out:
if (err)
goto out;
- err = pm_prepare(i915);
+ err = igt_pm_prepare(i915);
if (err)
goto out;
- pm_hibernate(i915);
+ igt_pm_hibernate(i915);
/* Here be dragons! */
simulate_hibernate(i915);
- pm_resume(i915);
+ igt_pm_resume(i915);
err = switch_to_context(ctx);
out:
if (err)
goto err_ppgtt_cleanup;
- err = i915_vm_pin_pt_stash(&ppgtt->vm, &stash);
+ err = i915_vm_map_pt_stash(&ppgtt->vm, &stash);
if (err) {
i915_vm_free_pt_stash(&ppgtt->vm, &stash);
goto err_ppgtt_cleanup;
if (err)
goto err_ppgtt_cleanup;
- err = i915_vm_pin_pt_stash(&ppgtt->vm, &stash);
+ err = i915_vm_map_pt_stash(&ppgtt->vm, &stash);
if (err) {
i915_vm_free_pt_stash(&ppgtt->vm, &stash);
goto err_ppgtt_cleanup;
BIT_ULL(size)))
goto alloc_vm_end;
- err = i915_vm_pin_pt_stash(vm, &stash);
+ err = i915_vm_map_pt_stash(vm, &stash);
if (!err)
vm->allocate_va_range(vm, &stash,
addr, BIT_ULL(size));
-
i915_vm_free_pt_stash(vm, &stash);
alloc_vm_end:
if (err == -EDEADLK) {
if (err)
goto end_ww;
- err = i915_vm_pin_pt_stash(vm, &stash);
+ err = i915_vm_map_pt_stash(vm, &stash);
if (!err)
vm->allocate_va_range(vm, &stash, offset, chunk_size);
-
i915_vm_free_pt_stash(vm, &stash);
end_ww:
if (err == -EDEADLK) {
}
/* Poison the ce->vm so we detect writes not to the GGTT gt->scratch */
- scratch = kmap(__px_page(ce->vm->scratch[0]));
+ scratch = __px_vaddr(ce->vm->scratch[0]);
memset(scratch, POISON_FREE, PAGE_SIZE);
rq = intel_context_create_request(ce);
out_rq:
i915_request_put(rq);
out_ce:
- kunmap(__px_page(ce->vm->scratch[0]));
intel_context_put(ce);
out:
stream_destroy(stream);
intel_wakeref_t wakeref;
int err = 0;
+ if (!i915_ggtt_has_aperture(&i915->ggtt))
+ return 0;
+
obj = i915_gem_object_create_internal(i915, 10 * 10 * PAGE_SIZE);
if (IS_ERR(obj))
return PTR_ERR(obj);
}
if (!spin->batch) {
- unsigned int mode =
- i915_coherent_map_type(spin->gt->i915);
+ unsigned int mode;
+ mode = i915_coherent_map_type(spin->gt->i915, spin->obj, false);
vaddr = igt_spinner_pin_obj(ce, ww, spin->obj, mode, &spin->batch_vma);
if (IS_ERR(vaddr))
return PTR_ERR(vaddr);
if (err)
return err;
- ptr = i915_gem_object_pin_map_unlocked(obj, I915_MAP_WC);
+ ptr = i915_gem_object_pin_map(obj, I915_MAP_WC);
if (IS_ERR(ptr))
return PTR_ERR(ptr);
if (err)
break;
+ i915_gem_object_lock(obj, NULL);
err = igt_cpu_check(obj, dword, rng);
+ i915_gem_object_unlock(obj);
if (err)
break;
} while (!__igt_timeout(end_time, NULL));
return err;
}
+static int igt_lmem_create_cleared_cpu(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ I915_RND_STATE(prng);
+ IGT_TIMEOUT(end_time);
+ u32 size, i;
+ int err;
+
+ i915_gem_drain_freed_objects(i915);
+
+ size = max_t(u32, PAGE_SIZE, i915_prandom_u32_max_state(SZ_32M, &prng));
+ size = round_up(size, PAGE_SIZE);
+ i = 0;
+
+ do {
+ struct drm_i915_gem_object *obj;
+ unsigned int flags;
+ u32 dword, val;
+ void *vaddr;
+
+ /*
+ * Alternate between cleared and uncleared allocations, while
+ * also dirtying the pages each time to check that the pages are
+ * always cleared if requested, since we should get some overlap
+ * of the underlying pages, if not all, since we are the only
+ * user.
+ */
+
+ flags = I915_BO_ALLOC_CPU_CLEAR;
+ if (i & 1)
+ flags = 0;
+
+ obj = i915_gem_object_create_lmem(i915, size, flags);
+ if (IS_ERR(obj))
+ return PTR_ERR(obj);
+
+ i915_gem_object_lock(obj, NULL);
+ err = i915_gem_object_pin_pages(obj);
+ if (err)
+ goto out_put;
+
+ dword = i915_prandom_u32_max_state(PAGE_SIZE / sizeof(u32),
+ &prng);
+
+ if (flags & I915_BO_ALLOC_CPU_CLEAR) {
+ err = igt_cpu_check(obj, dword, 0);
+ if (err) {
+ pr_err("%s failed with size=%u, flags=%u\n",
+ __func__, size, flags);
+ goto out_unpin;
+ }
+ }
+
+ vaddr = i915_gem_object_pin_map(obj, I915_MAP_WC);
+ if (IS_ERR(vaddr)) {
+ err = PTR_ERR(vaddr);
+ goto out_unpin;
+ }
+
+ val = prandom_u32_state(&prng);
+
+ memset32(vaddr, val, obj->base.size / sizeof(u32));
+
+ i915_gem_object_flush_map(obj);
+ i915_gem_object_unpin_map(obj);
+out_unpin:
+ i915_gem_object_unpin_pages(obj);
+ __i915_gem_object_put_pages(obj);
+out_put:
+ i915_gem_object_unlock(obj);
+ i915_gem_object_put(obj);
+
+ if (err)
+ break;
+ ++i;
+ } while (!__igt_timeout(end_time, NULL));
+
+ pr_info("%s completed (%u) iterations\n", __func__, i);
+
+ return err;
+}
+
static int igt_lmem_write_gpu(void *arg)
{
struct drm_i915_private *i915 = arg;
{
static const struct i915_subtest tests[] = {
SUBTEST(igt_lmem_create),
+ SUBTEST(igt_lmem_create_cleared_cpu),
SUBTEST(igt_lmem_write_cpu),
SUBTEST(igt_lmem_write_gpu),
};
#include <asm/msr.h>
+#include "i915_drv.h"
#include "librapl.h"
+bool librapl_supported(const struct drm_i915_private *i915)
+{
+ /* Discrete cards require hwmon integration */
+ if (IS_DGFX(i915))
+ return false;
+
+ return librapl_energy_uJ();
+}
+
u64 librapl_energy_uJ(void)
{
unsigned long long power;
#include <linux/types.h>
+struct drm_i915_private;
+
+bool librapl_supported(const struct drm_i915_private *i915);
+
u64 librapl_energy_uJ(void);
#endif /* SELFTEST_LIBRAPL_H */
#define I915_ERROR_UEVENT "ERROR"
#define I915_RESET_UEVENT "RESET"
-/*
- * i915_user_extension: Base class for defining a chain of extensions
+/**
+ * struct i915_user_extension - Base class for defining a chain of extensions
*
* Many interfaces need to grow over time. In most cases we can simply
* extend the struct and have userspace pass in more data. Another option,
* increasing complexity, and for large parts of that interface to be
* entirely optional. The downside is more pointer chasing; chasing across
* the __user boundary with pointers encapsulated inside u64.
+ *
+ * Example chaining:
+ *
+ * .. code-block:: C
+ *
+ * struct i915_user_extension ext3 {
+ * .next_extension = 0, // end
+ * .name = ...,
+ * };
+ * struct i915_user_extension ext2 {
+ * .next_extension = (uintptr_t)&ext3,
+ * .name = ...,
+ * };
+ * struct i915_user_extension ext1 {
+ * .next_extension = (uintptr_t)&ext2,
+ * .name = ...,
+ * };
+ *
+ * Typically the struct i915_user_extension would be embedded in some uAPI
+ * struct, and in this case we would feed it the head of the chain(i.e ext1),
+ * which would then apply all of the above extensions.
+ *
*/
struct i915_user_extension {
+ /**
+ * @next_extension:
+ *
+ * Pointer to the next struct i915_user_extension, or zero if the end.
+ */
__u64 next_extension;
+ /**
+ * @name: Name of the extension.
+ *
+ * Note that the name here is just some integer.
+ *
+ * Also note that the name space for this is not global for the whole
+ * driver, but rather its scope/meaning is limited to the specific piece
+ * of uAPI which has embedded the struct i915_user_extension.
+ */
__u32 name;
- __u32 flags; /* All undefined bits must be zero. */
- __u32 rsvd[4]; /* Reserved for future use; must be zero. */
+ /**
+ * @flags: MBZ
+ *
+ * All undefined bits must be zero.
+ */
+ __u32 flags;
+ /**
+ * @rsvd: MBZ
+ *
+ * Reserved for future use; must be zero.
+ */
+ __u32 rsvd[4];
};
/*
#define DRM_I915_QUERY 0x39
#define DRM_I915_GEM_VM_CREATE 0x3a
#define DRM_I915_GEM_VM_DESTROY 0x3b
+#define DRM_I915_GEM_CREATE_EXT 0x3c
/* Must be kept compact -- no holes */
#define DRM_IOCTL_I915_INIT DRM_IOW( DRM_COMMAND_BASE + DRM_I915_INIT, drm_i915_init_t)
#define DRM_IOCTL_I915_GEM_ENTERVT DRM_IO(DRM_COMMAND_BASE + DRM_I915_GEM_ENTERVT)
#define DRM_IOCTL_I915_GEM_LEAVEVT DRM_IO(DRM_COMMAND_BASE + DRM_I915_GEM_LEAVEVT)
#define DRM_IOCTL_I915_GEM_CREATE DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_CREATE, struct drm_i915_gem_create)
+#define DRM_IOCTL_I915_GEM_CREATE_EXT DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_CREATE_EXT, struct drm_i915_gem_create_ext)
#define DRM_IOCTL_I915_GEM_PREAD DRM_IOW (DRM_COMMAND_BASE + DRM_I915_GEM_PREAD, struct drm_i915_gem_pread)
#define DRM_IOCTL_I915_GEM_PWRITE DRM_IOW (DRM_COMMAND_BASE + DRM_I915_GEM_PWRITE, struct drm_i915_gem_pwrite)
#define DRM_IOCTL_I915_GEM_MMAP DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_MMAP, struct drm_i915_gem_mmap)
__u32 flags;
};
-/**
+/*
* See drm_i915_gem_execbuffer_ext_timeline_fences.
*/
#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
-/**
+/*
* This structure describes an array of drm_syncobj and associated points for
* timeline variants of drm_syncobj. It is invalid to append this structure to
* the execbuf if I915_EXEC_FENCE_ARRAY is set.
__u64 value;
};
-/**
+/*
* Context SSEU programming
*
* It may be necessary for either functional or performance reason to configure
__u64 properties_ptr;
};
-/**
+/*
* Enable data capture for a stream that was either opened in a disabled state
* via I915_PERF_FLAG_DISABLED or was later disabled via
* I915_PERF_IOCTL_DISABLE.
*/
#define I915_PERF_IOCTL_ENABLE _IO('i', 0x0)
-/**
+/*
* Disable data capture for a stream.
*
* It is an error to try and read a stream that is disabled.
*/
#define I915_PERF_IOCTL_DISABLE _IO('i', 0x1)
-/**
+/*
* Change metrics_set captured by a stream.
*
* If the stream is bound to a specific context, the configuration change
*/
#define I915_PERF_IOCTL_CONFIG _IO('i', 0x2)
-/**
+/*
* Common to all i915 perf records
*/
struct drm_i915_perf_record_header {
DRM_I915_PERF_RECORD_MAX /* non-ABI */
};
-/**
+/*
* Structure to upload perf dynamic configuration into the kernel.
*/
struct drm_i915_perf_oa_config {
__u64 flex_regs_ptr;
};
+/**
+ * struct drm_i915_query_item - An individual query for the kernel to process.
+ *
+ * The behaviour is determined by the @query_id. Note that exactly what
+ * @data_ptr is also depends on the specific @query_id.
+ */
struct drm_i915_query_item {
+ /** @query_id: The id for this query */
__u64 query_id;
#define DRM_I915_QUERY_TOPOLOGY_INFO 1
#define DRM_I915_QUERY_ENGINE_INFO 2
#define DRM_I915_QUERY_PERF_CONFIG 3
+#define DRM_I915_QUERY_MEMORY_REGIONS 4
/* Must be kept compact -- no holes and well documented */
- /*
+ /**
+ * @length:
+ *
* When set to zero by userspace, this is filled with the size of the
- * data to be written at the data_ptr pointer. The kernel sets this
+ * data to be written at the @data_ptr pointer. The kernel sets this
* value to a negative value to signal an error on a particular query
* item.
*/
__s32 length;
- /*
+ /**
+ * @flags:
+ *
* When query_id == DRM_I915_QUERY_TOPOLOGY_INFO, must be 0.
*
* When query_id == DRM_I915_QUERY_PERF_CONFIG, must be one of the
- * following :
- * - DRM_I915_QUERY_PERF_CONFIG_LIST
- * - DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID
- * - DRM_I915_QUERY_PERF_CONFIG_FOR_UUID
+ * following:
+ *
+ * - DRM_I915_QUERY_PERF_CONFIG_LIST
+ * - DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID
+ * - DRM_I915_QUERY_PERF_CONFIG_FOR_UUID
*/
__u32 flags;
#define DRM_I915_QUERY_PERF_CONFIG_LIST 1
#define DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_UUID 2
#define DRM_I915_QUERY_PERF_CONFIG_DATA_FOR_ID 3
- /*
- * Data will be written at the location pointed by data_ptr when the
- * value of length matches the length of the data to be written by the
+ /**
+ * @data_ptr:
+ *
+ * Data will be written at the location pointed by @data_ptr when the
+ * value of @length matches the length of the data to be written by the
* kernel.
*/
__u64 data_ptr;
};
+/**
+ * struct drm_i915_query - Supply an array of struct drm_i915_query_item for the
+ * kernel to fill out.
+ *
+ * Note that this is generally a two step process for each struct
+ * drm_i915_query_item in the array:
+ *
+ * 1. Call the DRM_IOCTL_I915_QUERY, giving it our array of struct
+ * drm_i915_query_item, with &drm_i915_query_item.length set to zero. The
+ * kernel will then fill in the size, in bytes, which tells userspace how
+ * memory it needs to allocate for the blob(say for an array of properties).
+ *
+ * 2. Next we call DRM_IOCTL_I915_QUERY again, this time with the
+ * &drm_i915_query_item.data_ptr equal to our newly allocated blob. Note that
+ * the &drm_i915_query_item.length should still be the same as what the
+ * kernel previously set. At this point the kernel can fill in the blob.
+ *
+ * Note that for some query items it can make sense for userspace to just pass
+ * in a buffer/blob equal to or larger than the required size. In this case only
+ * a single ioctl call is needed. For some smaller query items this can work
+ * quite well.
+ *
+ */
struct drm_i915_query {
+ /** @num_items: The number of elements in the @items_ptr array */
__u32 num_items;
- /*
- * Unused for now. Must be cleared to zero.
+ /**
+ * @flags: Unused for now. Must be cleared to zero.
*/
__u32 flags;
- /*
- * This points to an array of num_items drm_i915_query_item structures.
+ /**
+ * @items_ptr:
+ *
+ * Pointer to an array of struct drm_i915_query_item. The number of
+ * array elements is @num_items.
*/
__u64 items_ptr;
};
* Describes one engine and it's capabilities as known to the driver.
*/
struct drm_i915_engine_info {
- /** Engine class and instance. */
+ /** @engine: Engine class and instance. */
struct i915_engine_class_instance engine;
- /** Reserved field. */
+ /** @rsvd0: Reserved field. */
__u32 rsvd0;
- /** Engine flags. */
+ /** @flags: Engine flags. */
__u64 flags;
- /** Capabilities of this engine. */
+ /** @capabilities: Capabilities of this engine. */
__u64 capabilities;
#define I915_VIDEO_CLASS_CAPABILITY_HEVC (1 << 0)
#define I915_VIDEO_AND_ENHANCE_CLASS_CAPABILITY_SFC (1 << 1)
- /** Reserved fields. */
+ /** @rsvd1: Reserved fields. */
__u64 rsvd1[4];
};
* an array of struct drm_i915_engine_info structures.
*/
struct drm_i915_query_engine_info {
- /** Number of struct drm_i915_engine_info structs following. */
+ /** @num_engines: Number of struct drm_i915_engine_info structs following. */
__u32 num_engines;
- /** MBZ */
+ /** @rsvd: MBZ */
__u32 rsvd[3];
- /** Marker for drm_i915_engine_info structures. */
+ /** @engines: Marker for drm_i915_engine_info structures. */
struct drm_i915_engine_info engines[];
};
__u8 data[];
};
+/**
+ * enum drm_i915_gem_memory_class - Supported memory classes
+ */
+enum drm_i915_gem_memory_class {
+ /** @I915_MEMORY_CLASS_SYSTEM: System memory */
+ I915_MEMORY_CLASS_SYSTEM = 0,
+ /** @I915_MEMORY_CLASS_DEVICE: Device local-memory */
+ I915_MEMORY_CLASS_DEVICE,
+};
+
+/**
+ * struct drm_i915_gem_memory_class_instance - Identify particular memory region
+ */
+struct drm_i915_gem_memory_class_instance {
+ /** @memory_class: See enum drm_i915_gem_memory_class */
+ __u16 memory_class;
+
+ /** @memory_instance: Which instance */
+ __u16 memory_instance;
+};
+
+/**
+ * struct drm_i915_memory_region_info - Describes one region as known to the
+ * driver.
+ *
+ * Note that we reserve some stuff here for potential future work. As an example
+ * we might want expose the capabilities for a given region, which could include
+ * things like if the region is CPU mappable/accessible, what are the supported
+ * mapping types etc.
+ *
+ * Note that to extend struct drm_i915_memory_region_info and struct
+ * drm_i915_query_memory_regions in the future the plan is to do the following:
+ *
+ * .. code-block:: C
+ *
+ * struct drm_i915_memory_region_info {
+ * struct drm_i915_gem_memory_class_instance region;
+ * union {
+ * __u32 rsvd0;
+ * __u32 new_thing1;
+ * };
+ * ...
+ * union {
+ * __u64 rsvd1[8];
+ * struct {
+ * __u64 new_thing2;
+ * __u64 new_thing3;
+ * ...
+ * };
+ * };
+ * };
+ *
+ * With this things should remain source compatible between versions for
+ * userspace, even as we add new fields.
+ *
+ * Note this is using both struct drm_i915_query_item and struct drm_i915_query.
+ * For this new query we are adding the new query id DRM_I915_QUERY_MEMORY_REGIONS
+ * at &drm_i915_query_item.query_id.
+ */
+struct drm_i915_memory_region_info {
+ /** @region: The class:instance pair encoding */
+ struct drm_i915_gem_memory_class_instance region;
+
+ /** @rsvd0: MBZ */
+ __u32 rsvd0;
+
+ /** @probed_size: Memory probed by the driver (-1 = unknown) */
+ __u64 probed_size;
+
+ /** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
+ __u64 unallocated_size;
+
+ /** @rsvd1: MBZ */
+ __u64 rsvd1[8];
+};
+
+/**
+ * struct drm_i915_query_memory_regions
+ *
+ * The region info query enumerates all regions known to the driver by filling
+ * in an array of struct drm_i915_memory_region_info structures.
+ *
+ * Example for getting the list of supported regions:
+ *
+ * .. code-block:: C
+ *
+ * struct drm_i915_query_memory_regions *info;
+ * struct drm_i915_query_item item = {
+ * .query_id = DRM_I915_QUERY_MEMORY_REGIONS;
+ * };
+ * struct drm_i915_query query = {
+ * .num_items = 1,
+ * .items_ptr = (uintptr_t)&item,
+ * };
+ * int err, i;
+ *
+ * // First query the size of the blob we need, this needs to be large
+ * // enough to hold our array of regions. The kernel will fill out the
+ * // item.length for us, which is the number of bytes we need.
+ * err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
+ * if (err) ...
+ *
+ * info = calloc(1, item.length);
+ * // Now that we allocated the required number of bytes, we call the ioctl
+ * // again, this time with the data_ptr pointing to our newly allocated
+ * // blob, which the kernel can then populate with the all the region info.
+ * item.data_ptr = (uintptr_t)&info,
+ *
+ * err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
+ * if (err) ...
+ *
+ * // We can now access each region in the array
+ * for (i = 0; i < info->num_regions; i++) {
+ * struct drm_i915_memory_region_info mr = info->regions[i];
+ * u16 class = mr.region.class;
+ * u16 instance = mr.region.instance;
+ *
+ * ....
+ * }
+ *
+ * free(info);
+ */
+struct drm_i915_query_memory_regions {
+ /** @num_regions: Number of supported regions */
+ __u32 num_regions;
+
+ /** @rsvd: MBZ */
+ __u32 rsvd[3];
+
+ /** @regions: Info about each supported region */
+ struct drm_i915_memory_region_info regions[];
+};
+
+/**
+ * struct drm_i915_gem_create_ext - Existing gem_create behaviour, with added
+ * extension support using struct i915_user_extension.
+ *
+ * Note that in the future we want to have our buffer flags here, at least for
+ * the stuff that is immutable. Previously we would have two ioctls, one to
+ * create the object with gem_create, and another to apply various parameters,
+ * however this creates some ambiguity for the params which are considered
+ * immutable. Also in general we're phasing out the various SET/GET ioctls.
+ */
+struct drm_i915_gem_create_ext {
+ /**
+ * @size: Requested size for the object.
+ *
+ * The (page-aligned) allocated size for the object will be returned.
+ *
+ * Note that for some devices we have might have further minimum
+ * page-size restrictions(larger than 4K), like for device local-memory.
+ * However in general the final size here should always reflect any
+ * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
+ * extension to place the object in device local-memory.
+ */
+ __u64 size;
+ /**
+ * @handle: Returned handle for the object.
+ *
+ * Object handles are nonzero.
+ */
+ __u32 handle;
+ /** @flags: MBZ */
+ __u32 flags;
+ /**
+ * @extensions: The chain of extensions to apply to this object.
+ *
+ * This will be useful in the future when we need to support several
+ * different extensions, and we need to apply more than one when
+ * creating the object. See struct i915_user_extension.
+ *
+ * If we don't supply any extensions then we get the same old gem_create
+ * behaviour.
+ *
+ * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
+ * struct drm_i915_gem_create_ext_memory_regions.
+ */
+#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
+ __u64 extensions;
+};
+
+/**
+ * struct drm_i915_gem_create_ext_memory_regions - The
+ * I915_GEM_CREATE_EXT_MEMORY_REGIONS extension.
+ *
+ * Set the object with the desired set of placements/regions in priority
+ * order. Each entry must be unique and supported by the device.
+ *
+ * This is provided as an array of struct drm_i915_gem_memory_class_instance, or
+ * an equivalent layout of class:instance pair encodings. See struct
+ * drm_i915_query_memory_regions and DRM_I915_QUERY_MEMORY_REGIONS for how to
+ * query the supported regions for a device.
+ *
+ * As an example, on discrete devices, if we wish to set the placement as
+ * device local-memory we can do something like:
+ *
+ * .. code-block:: C
+ *
+ * struct drm_i915_gem_memory_class_instance region_lmem = {
+ * .memory_class = I915_MEMORY_CLASS_DEVICE,
+ * .memory_instance = 0,
+ * };
+ * struct drm_i915_gem_create_ext_memory_regions regions = {
+ * .base = { .name = I915_GEM_CREATE_EXT_MEMORY_REGIONS },
+ * .regions = (uintptr_t)®ion_lmem,
+ * .num_regions = 1,
+ * };
+ * struct drm_i915_gem_create_ext create_ext = {
+ * .size = 16 * PAGE_SIZE,
+ * .extensions = (uintptr_t)®ions,
+ * };
+ *
+ * int err = ioctl(fd, DRM_IOCTL_I915_GEM_CREATE_EXT, &create_ext);
+ * if (err) ...
+ *
+ * At which point we get the object handle in &drm_i915_gem_create_ext.handle,
+ * along with the final object size in &drm_i915_gem_create_ext.size, which
+ * should account for any rounding up, if required.
+ */
+struct drm_i915_gem_create_ext_memory_regions {
+ /** @base: Extension link. See struct i915_user_extension. */
+ struct i915_user_extension base;
+
+ /** @pad: MBZ */
+ __u32 pad;
+ /** @num_regions: Number of elements in the @regions array. */
+ __u32 num_regions;
+ /**
+ * @regions: The regions/placements array.
+ *
+ * An array of struct drm_i915_gem_memory_class_instance.
+ */
+ __u64 regions;
+};
+
#if defined(__cplusplus)
}
#endif
projects / linux-2.6-microblaze.git / commitdiff