Add a page explaining IOMMU to Conceptual section (#3851)

* Add a page explaining IOMMU to Conceptual section

* fix typo, expand wordlist

* fix typo

* streamlined the iommu section (#3947)

* streamlined the iommu section

* made changes based on feedback

* tweaked some language around xgmi etc

* removed the reference to the MI300 to ensure a long shelf-life for this content

* changed onwards bc the spellcheck didn't like it

* Update docs/conceptual/iommu.rst

Co-authored-by: Leo Paoletti <164940351+lpaoletti@users.noreply.github.com>

* Update docs/conceptual/iommu.rst

Co-authored-by: Leo Paoletti <164940351+lpaoletti@users.noreply.github.com>

* Update docs/conceptual/iommu.rst

Co-authored-by: Leo Paoletti <164940351+lpaoletti@users.noreply.github.com>

* Update docs/conceptual/iommu.rst

Co-authored-by: Leo Paoletti <164940351+lpaoletti@users.noreply.github.com>

* Update docs/conceptual/iommu.rst

Co-authored-by: Leo Paoletti <164940351+lpaoletti@users.noreply.github.com>

* Updated based

---------

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>
Co-authored-by: Leo Paoletti <164940351+lpaoletti@users.noreply.github.com>

docs/conceptual/iommu.rst

@@ -0,0 +1,55 @@
+.. meta::
+   :description: Input-Output Memory Management Unit (IOMMU)
+   :keywords: IOMMU, DMA, PCIe, xGMI, AMD, ROCm
+
+****************************************************************
+Input-Output Memory Management Unit (IOMMU)
+****************************************************************
+
+The I/O Memory Management Unit (IOMMU) provides memory remapping services for I/O devices. It adds support for address translation and system memory access protection on direct memory access (DMA) transfers from peripheral devices. 
+
+The IOMMU's memory remapping services:
+
+* provide private I/O space for devices used in a guest virtual machine.
+* prevent unauthorized DMA requests to system memory and to memory-mapped I/O (MMIO).
+* help in debugging memory access issues.
+* facilitate peer-to-peer DMA.
+
+The IOMMU also provides interrupt remapping, which is used by devices that support multiple interrupts and for interrupt delivery on hardware platforms with a large number of cores.
+
+.. note::
+
+  AMD Instinct accelerators are connected via XGMI links and don't use PCI/PCIe for peer-to-peer DMA. Because PCI/PCIe is not used for peer-to-peer DMA, there are no device physical addressing limitations or platform root port limitations. However, because non-GPU devices such as RDMA NICs use PCIe for peer-to-peer DMA, there might still be physical addressing and platform root port limitations when these non-GPU devices interact with other devices, including GPUs.
+
+Linux supports IOMMU in both virtualized environments and bare metal. 
+
+The IOMMU is enabled by default but can be disabled or put into passthrough mode through the Linux kernel command line:
+
+.. list-table:: 
+  :widths: 25 25 50
+  :header-rows: 1
+
+  * - IOMMU Mode
+    - Kernel command
+    - Description
+  * - Enabled
+    - default setting
+    - The IOMMU is enabled in remapping mode. Each device gets its own I/O virtual address space. All devices on Linux register their DMA addressing capabilities, and the kernel will ensure that any address space mapped for DMA is mapped within the device's DMA addressing limits. Only address space explicitly mapped by the devices will be mapped into virtual address space. Attempts to access an unmapped page will generate an IOMMU page fault. This setting is recommended for AMD Radeon GPUs that need peer-to-peer DMA.
+  * - Passthrough
+    - ``iommu=pt``
+    - Interrupt remapping is enabled but I/O remapping is disabled. The entire platform shares a common platform address space for system memory and MMIO spaces, ensuring compatibility with drivers from external vendors, while still supporting CPUs with a large number of cores. This setting is recommended for AMD Instinct Accelerators and for AMD Radeon GPUs that don't need peer-to-peer DMA.
+  * - Disabled
+    - ``iommu=off``
+    - The IOMMU is disabled and the entire platform shares a common platform address space for system memory and MMIO spaces.
+
+The IOMMU also provides virtualized access to the MMIO portions of the platform address space for peer-to-peer DMA.
+
+Because peer-to-peer DMA is not officially part of the PCI/PCIe specification, the behavior of peer-to-peer DMA varies between hardware platforms. 
+
+AMD CPUs earlier than AMD Zen only supported peer-to-peer DMA for writes. On CPUs from AMD Zen and later, peer-to-peer DMA is fully supported. 
+
+To use peer-to-peer DMA on Linux, enable the following options in your Linux kernel configuration:
+
+* ``CONFIG_PCI_P2PDMA``
+* ``CONFIG_DMABUF_MOVE_NOTIFY`` 
+* ``CONFIG_HSA_AMD_P2P``

docs/index.md

@@ -58,6 +58,7 @@ ROCm documentation is organized into the following categories:
 
 * [GPU architecture overview](./conceptual/gpu-arch.md)
 * [GPU memory](./conceptual/gpu-memory.md)
+* [Input-Output Memory Management Unit (IOMMU)](./conceptual/iommu.rst)
 * [File structure (Linux FHS)](./conceptual/file-reorg.md)
 * [GPU isolation techniques](./conceptual/gpu-isolation.md)
 * [Using CMake](./conceptual/cmake-packages.rst)

docs/sphinx/_toc.yml.in

@@ -152,6 +152,8 @@ subtrees:
             title: White paper
   - file: conceptual/gpu-memory.md
     title: GPU memory
+  - file: conceptual/iommu.rst
+    title: Input-Output Memory Management Unit (IOMMU)
   - file: conceptual/file-reorg.md
     title: File structure (Linux FHS)
   - file: conceptual/gpu-isolation.md