mirror of
https://github.com/ROCm/ROCm.git
synced 2026-02-04 11:25:03 -05:00
7b883f3af4
* Add MI300X tuning guides Add mi300x doc (pandoc conversion) fix headings add metadata move images to shared/ move images to shared/ convert tuning-guides.md to rst using pandoc add mi300x to tuning-guides.rst landing page update h1s, toc, and landing page fix spelling fix fmt format code blocks add tensilelite imgs fix formatting fix formatting some more fix formatting more formatting spelling remove --enforce-eager note satisfy spellcheck linter more spelling add fixes from hongxia fix env var in D5 add fixes to PyTorch inductor section fix fix Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update docs/how-to/tuning-guides/mi300x.rst Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update 'torch_compile_debug' suggestion based on Hongxia's feedback fix PyTorch inductor env vars minor formatting fixes Apply suggestions from code review Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> Update vllm path Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> disable numfig in Sphinx configuration fix formatting and capitalization add words to wordlist update index update wordlist update optimizing-triton-kernel convert cards to table fix link in index.md add @lpaoletti's feedback Add system tuning guide add images add system section add os settings and sys management remove pcie=noats recommendation reorg add blurb to developer section impr formatting remove windows os from tuning guides pages in conf.py add suggestions from review fix typo and link remove os windows from relevant pages in conf mi300x add suggestions from review fix toc fix index links reorg update vLLM vars Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> update vLLM vars Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com> reorganize add warnings add text to system tuning add filler text on index pages reorg tuning pages fix links fix vars * rm old pages fix toc * add suggestions from review small change add more suggestions rewrite intro * add 'workload tuning philosophy' * refactor * fix broken links * black format conf.py * simplify cmd and update doc structure * add higher-level heading for consistency (mi300x.rst) * add fixes from review fix url add fixes fix formatting fix fmt fix hipBLASLt section change words fix tensilelite section fix fix fix fmt * style guide * fix some formatting * satisfy spellcheck linter * update wordlist * fix bad conflict resolution
566 lines
13 KiB
ReStructuredText
566 lines
13 KiB
ReStructuredText
.. meta::
|
|
:description: Supported data types in ROCm
|
|
:keywords: int8, float8, float8 (E4M3), float8 (E5M2), bfloat8, float16, half, bfloat16, tensorfloat32, float,
|
|
float32, float64, double, AMD, ROCm, AMDGPU
|
|
|
|
*************************************************************
|
|
Precision support
|
|
*************************************************************
|
|
|
|
Use the following sections to identify data types and HIP types ROCm™ supports.
|
|
|
|
Integral types
|
|
==========================================
|
|
|
|
The signed and unsigned integral types that are supported by ROCm are listed in the following table,
|
|
together with their corresponding HIP type and a short description.
|
|
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 15,35,50
|
|
|
|
*
|
|
- Type name
|
|
- HIP type
|
|
- Description
|
|
*
|
|
- int8
|
|
- ``int8_t``, ``uint8_t``
|
|
- A signed or unsigned 8-bit integer
|
|
*
|
|
- int16
|
|
- ``int16_t``, ``uint16_t``
|
|
- A signed or unsigned 16-bit integer
|
|
*
|
|
- int32
|
|
- ``int32_t``, ``uint32_t``
|
|
- A signed or unsigned 32-bit integer
|
|
*
|
|
- int64
|
|
- ``int64_t``, ``uint64_t``
|
|
- A signed or unsigned 64-bit integer
|
|
|
|
Floating-point types
|
|
==========================================
|
|
|
|
The floating-point types that are supported by ROCm are listed in the following table, together with
|
|
their corresponding HIP type and a short description.
|
|
|
|
.. image:: ../data/about/compatibility/floating-point-data-types.png
|
|
:alt: Supported floating-point types
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 15,15,70
|
|
|
|
*
|
|
- Type name
|
|
- HIP type
|
|
- Description
|
|
*
|
|
- float8 (E4M3)
|
|
- ``-``
|
|
- An 8-bit floating-point number that mostly follows IEEE-754 conventions and **S1E4M3** bit layout, as described in `8-bit Numerical Formats for Deep Neural Networks <https://arxiv.org/abs/2206.02915>`_ , with expanded range and with no infinity or signed zero. NaN is represented as negative zero.
|
|
*
|
|
- float8 (E5M2)
|
|
- ``-``
|
|
- An 8-bit floating-point number mostly following IEEE-754 conventions and **S1E5M2** bit layout, as described in `8-bit Numerical Formats for Deep Neural Networks <https://arxiv.org/abs/2206.02915>`_ , with expanded range and with no infinity or signed zero. NaN is represented as negative zero.
|
|
*
|
|
- float16
|
|
- ``half``
|
|
- A 16-bit floating-point number that conforms to the IEEE 754-2008 half-precision storage format.
|
|
*
|
|
- bfloat16
|
|
- ``bfloat16``
|
|
- A shortened 16-bit version of the IEEE 754 single-precision storage format.
|
|
*
|
|
- tensorfloat32
|
|
- ``-``
|
|
- A floating-point number that occupies 32 bits or less of storage, providing improved range compared to half (16-bit) format, at (potentially) greater throughput than single-precision (32-bit) formats.
|
|
*
|
|
- float32
|
|
- ``float``
|
|
- A 32-bit floating-point number that conforms to the IEEE 754 single-precision storage format.
|
|
*
|
|
- float64
|
|
- ``double``
|
|
- A 64-bit floating-point number that conforms to the IEEE 754 double-precision storage format.
|
|
|
|
.. note::
|
|
|
|
* The float8 and tensorfloat32 types are internal types used in calculations in Matrix Cores and can be stored in any type of the same size.
|
|
* The encodings for FP8 (E5M2) and FP8 (E4M3) that are natively supported by MI300 differ from the FP8 (E5M2) and FP8 (E4M3) encodings used in H100 (`FP8 Formats for Deep Learning <https://arxiv.org/abs/2209.05433>`_).
|
|
* In some AMD documents and articles, float8 (E5M2) is referred to as bfloat8.
|
|
|
|
ROCm support icons
|
|
==========================================
|
|
|
|
In the following sections, we use icons to represent the level of support. These icons, described in the
|
|
following table, are also used on the library data type support pages.
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Icon
|
|
- Definition
|
|
*
|
|
- ❌
|
|
- Not supported
|
|
|
|
*
|
|
- ⚠️
|
|
- Partial support
|
|
|
|
*
|
|
- ✅
|
|
- Full support
|
|
|
|
.. note::
|
|
|
|
* Full support means that the type is supported natively or with hardware emulation.
|
|
* Native support means that the operations for that type are implemented in hardware. Types that are not natively supported are emulated with the available hardware. The performance of non-natively supported types can differ from the full instruction throughput rate. For example, 16-bit integer operations can be performed on the 32-bit integer ALUs at full rate; however, 64-bit integer operations might need several instructions on the 32-bit integer ALUs.
|
|
* Any type can be emulated by software, but this page does not cover such cases.
|
|
|
|
Hardware type support
|
|
==========================================
|
|
|
|
AMD GPU hardware support for data types is listed in the following tables.
|
|
|
|
Compute units support
|
|
-------------------------------------------------------------------------------
|
|
|
|
The following table lists data type support for compute units.
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: Integral types
|
|
:sync: integral-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Type name
|
|
- int8
|
|
- int16
|
|
- int32
|
|
- int64
|
|
*
|
|
- MI100
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI200 series
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI300 series
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
|
|
.. tab-item:: Floating-point types
|
|
:sync: floating-point-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Type name
|
|
- float8 (E4M3)
|
|
- float8 (E5M2)
|
|
- float16
|
|
- bfloat16
|
|
- tensorfloat32
|
|
- float32
|
|
- float64
|
|
*
|
|
- MI100
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI200 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI300 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
|
|
Matrix core support
|
|
-------------------------------------------------------------------------------
|
|
|
|
The following table lists data type support for AMD GPU matrix cores.
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: Integral types
|
|
:sync: integral-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Type name
|
|
- int8
|
|
- int16
|
|
- int32
|
|
- int64
|
|
*
|
|
- MI100
|
|
- ✅
|
|
- ❌
|
|
- ❌
|
|
- ❌
|
|
*
|
|
- MI200 series
|
|
- ✅
|
|
- ❌
|
|
- ❌
|
|
- ❌
|
|
*
|
|
- MI300 series
|
|
- ✅
|
|
- ❌
|
|
- ❌
|
|
- ❌
|
|
|
|
.. tab-item:: Floating-point types
|
|
:sync: floating-point-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Type name
|
|
- float8 (E4M3)
|
|
- float8 (E5M2)
|
|
- float16
|
|
- bfloat16
|
|
- tensorfloat32
|
|
- float32
|
|
- float64
|
|
*
|
|
- MI100
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
*
|
|
- MI200 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI300 series
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
- ✅
|
|
|
|
Atomic operations support
|
|
-------------------------------------------------------------------------------
|
|
|
|
The following table lists data type support for atomic operations.
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: Integral types
|
|
:sync: integral-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Type name
|
|
- int8
|
|
- int16
|
|
- int32
|
|
- int64
|
|
*
|
|
- MI100
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
*
|
|
- MI200 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI300 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
|
|
.. tab-item:: Floating-point types
|
|
:sync: floating-point-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Type name
|
|
- float8 (E4M3)
|
|
- float8 (E5M2)
|
|
- float16
|
|
- bfloat16
|
|
- tensorfloat32
|
|
- float32
|
|
- float64
|
|
*
|
|
- MI100
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
*
|
|
- MI200 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
*
|
|
- MI300 series
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ✅
|
|
|
|
.. note::
|
|
|
|
For cases that are not natively supported, you can emulate atomic operations using software.
|
|
Software-emulated atomic operations have high negative performance impact when they frequently
|
|
access the same memory address.
|
|
|
|
Data Type support in ROCm Libraries
|
|
==========================================
|
|
|
|
ROCm library support for int8, float8 (E4M3), float8 (E5M2), int16, float16, bfloat16, int32,
|
|
tensorfloat32, float32, int64, and float64 is listed in the following tables.
|
|
|
|
Libraries input/output type support
|
|
-------------------------------------------------------------------------------
|
|
|
|
The following tables list ROCm library support for specific input and output data types. For a detailed
|
|
description, refer to the corresponding library data type support page.
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: Integral types
|
|
:sync: integral-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Library input/output data type name
|
|
- int8
|
|
- int16
|
|
- int32
|
|
- int64
|
|
*
|
|
- hipSPARSELt (:doc:`details <hipsparselt:reference/data-type-support>`)
|
|
- ✅/✅
|
|
- ❌/❌
|
|
- ❌/❌
|
|
- ❌/❌
|
|
*
|
|
- rocRAND (:doc:`details <rocrand:data-type-support>`)
|
|
- -/✅
|
|
- -/✅
|
|
- -/✅
|
|
- -/✅
|
|
*
|
|
- hipRAND (:doc:`details <hiprand:api-reference/data-type-support>`)
|
|
- -/✅
|
|
- -/✅
|
|
- -/✅
|
|
- -/✅
|
|
*
|
|
- rocPRIM (:doc:`details <rocprim:reference/data-type-support>`)
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ✅/✅
|
|
*
|
|
- hipCUB (:doc:`details <hipcub:api-reference/data-type-support>`)
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ✅/✅
|
|
*
|
|
- rocThrust (:doc:`details <rocthrust:data-type-support>`)
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ✅/✅
|
|
|
|
.. tab-item:: Floating-point types
|
|
:sync: floating-point-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Library input/output data type name
|
|
- float8 (E4M3)
|
|
- float8 (E5M2)
|
|
- float16
|
|
- bfloat16
|
|
- tensorfloat32
|
|
- float32
|
|
- float64
|
|
*
|
|
- hipSPARSELt (:doc:`details <hipsparselt:reference/data-type-support>`)
|
|
- ❌/❌
|
|
- ❌/❌
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ❌/❌
|
|
- ❌/❌
|
|
- ❌/❌
|
|
*
|
|
- rocRAND (:doc:`details <rocrand:data-type-support>`)
|
|
- -/❌
|
|
- -/❌
|
|
- -/✅
|
|
- -/❌
|
|
- -/❌
|
|
- -/✅
|
|
- -/✅
|
|
*
|
|
- hipRAND (:doc:`details <hiprand:api-reference/data-type-support>`)
|
|
- -/❌
|
|
- -/❌
|
|
- -/✅
|
|
- -/❌
|
|
- -/❌
|
|
- -/✅
|
|
- -/✅
|
|
*
|
|
- rocPRIM (:doc:`details <rocprim:reference/data-type-support>`)
|
|
- ❌/❌
|
|
- ❌/❌
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ❌/❌
|
|
- ✅/✅
|
|
- ✅/✅
|
|
*
|
|
- hipCUB (:doc:`details <hipcub:api-reference/data-type-support>`)
|
|
- ❌/❌
|
|
- ❌/❌
|
|
- ✅/✅
|
|
- ✅/✅
|
|
- ❌/❌
|
|
- ✅/✅
|
|
- ✅/✅
|
|
*
|
|
- rocThrust (:doc:`details <rocthrust:data-type-support>`)
|
|
- ❌/❌
|
|
- ❌/❌
|
|
- ⚠️/⚠️
|
|
- ⚠️/⚠️
|
|
- ❌/❌
|
|
- ✅/✅
|
|
- ✅/✅
|
|
|
|
|
|
Libraries internal calculations type support
|
|
-------------------------------------------------------------------------------
|
|
|
|
The following tables list ROCm library support for specific internal data types. For a detailed
|
|
description, refer to the corresponding library data type support page.
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: Integral types
|
|
:sync: integral-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Library internal data type name
|
|
- int8
|
|
- int16
|
|
- int32
|
|
- int64
|
|
*
|
|
- hipSPARSELt (:doc:`details <hipsparselt:reference/data-type-support>`)
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|
|
|
|
|
|
.. tab-item:: Floating-point types
|
|
:sync: floating-point-type
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
|
|
*
|
|
- Library internal data type name
|
|
- float8 (E4M3)
|
|
- float8 (E5M2)
|
|
- float16
|
|
- bfloat16
|
|
- tensorfloat32
|
|
- float32
|
|
- float64
|
|
*
|
|
- hipSPARSELt (:doc:`details <hipsparselt:reference/data-type-support>`)
|
|
- ❌
|
|
- ❌
|
|
- ❌
|
|
- ❌
|
|
- ❌
|
|
- ✅
|
|
- ❌
|