Covered in the NVMe 1.4 eLearning course (included)

• PCIe Architecture Overview
  • Introduction
  • Basic operation
  • Transaction types
  • Layers and responsibilities overview
• NVMe Introduction
  • NVMe concepts and definitions
  • Comparison with SATA
  • NVMe Host-Controller Interface overview
  • Queue Management
    • Doorbell Head and Tail registers
    • Execution sequence
    • Tracking completion status
• Commands
  • NVMe Admin Commands
  • General Structure of commands
  • Identify command options and results
  • Creation and management of Queues
  • Asynchronous Event Notification
  • NVMe IO Commands
    • Read and Write command structure
    • Optimizing write commands
    • Avoiding write amplification
    • Buffer addressing modes (PRPs vs. SGLs)
    • Priority and Arbitration of commands
    • Completing commands
    • Handling Metadata (DIF/DIX)
    • Get Features / Set Features – full coverage of features

• Architecture
  • Error reporting, Error Reporting Structures
    • Asynchronous Events
    • Completion status
    • Log pages
  • Other log pages
  • Firmware updates
  • Controller registers
  • Controller initialization
  • Power Management
  • Reservations
•  Other NVMe commands
  • Write uncorrectable
  • Write zeroes
  • Compare
  • Dataset management
  • Configuration overview
• Summary of changes for 1.3
  • Boot partitions
  • Thermal management by host
  • Write streams
  • Deallocation
  • Virtualization support
  • Debug support – Telemetry, initiated by host or device
  • Self-test options
  • Sanitize command
  • New commands
• Summary of changes for 1.4
  • NVM Sets
  • I/O Determinism
  • Read recovery levels
  • Persistent Memory Region (PMR)
  • Asymmetric Namespace Access (ANA)
  • Namespace write protect
  • Persistent event log
  • Rebuild assist
  • Endurance groups
  • I/O Performance and Endurance Hints
  • Traffic-based Keep Alive
  • Vendor-specific UUIDs
  • Verify - new command
  • Administrative controller
  • SQ Associations

Covered in the NVMe 2.x eLearning course (included)

• Summary of select updates from 2.0 through 2.3 specs
  • Simple copy command 
  • Domains and Partitioning
  • NVM Set and Endurance Group Management
  • Command and Feature Lockdown
  • NVMe KV
  • Zoned Namespaces
  •  Namespace types
  • IO Command Set Independent Data Structure
  • Rotational media support
  • Command and Effect Log (CEL) Enhancements
  • Non-MDT Command Size Limits
  • CMB Write Elasticity Status
  • Multiple Controller Firmware Update
  •  Telemetry Log Size Changes
  • Telemetry Enhancements
  • Namespace Attachment Limits
  • Time To Ready Enhancements
  • Graceful Disconnection
  • Persistent Event Log Context Counter
  •  SGL Optimizations
  • Protection Information Enhancement
  • ZNS Status Code and Async Notification
  •  Aborting Commands During Format NVM Command
  • Multiple Controller Shutdown Enhancement
  •  Persistent Event Log (PEL) State and Source Reporting
  • Get Log Page Index Offsets
  • In-band Authentication Protocol for NVMe over Fabrics
  • Transport Layer Security (TLS) 1.3 Support
  • Set/Get Features Ported from NVMe-MI to the NVMe 2.0 Base Spec
  •  Boot Partition Read Access using OOB
  • Additional Controller Metadata Types

Covered in the NVMe-oF eLearning course (included)

• Introduction
  • Motivation for moving NVMe to fabrics
    • Improve storage network performance
    • Avoid limitations of PCIe, extend NVMe protocol to the fabric
    • Optimize access latency across the network
  • Differences from base NVMe
  • Initialization Background
• Fabric Discovery
  • Discovery Service, Discovery Controllers
  • Discovery Mechanism
• NVMe-oF Initialization
  • Fabrics Commands
  • Device Setup
  • Transport mapping/binding RDMA
  • Introduction to RDMA (Remote Direct Memory Access)
  • Protocol Translation
  • RDMA Examples
• RDMA Concepts
  • Motivation: Kernel bypass – reduce CPU involvement
  • Zero-copy operation when possible
  • RNIC (RDMA-aware Network Interface Card)
  • Pinned remote memory and storage
  • Command-based Flow Control
  • Read and Write Command Examples
• NVMe-oF Details
  • Controller Initialization
  • Connections and Associations
  • Static vs. Dynamic Controller model
  • Properties instead of registers
  • Authentication
  • Encapsulation
  • Accessing RDMA Queue pairs
  • Data transfer: Command, Response, Data, Flow Control
  • Fabrics Commands and Responses
• RNIC Details
  • Software Layers
  • RDMA Verbs
  • Memory Regions and Windows
• Ethernet overview
  • TCP/IP: addressing, layers, packets
  • RDMA models added to it: RoCE and iWARP
    • Overview of iWARP
      • Added to TCP/IP, so packets are routable and don’t need DCB
      • Messages
      • Layers
      • Advantages and disadvantages
    • Overview of RoCE
      • Version 2 packets include routing information, version 1 did not
      • Use of Data Center Bridging (DCB) to manage routing
      • Flow control
      • Soft RoCE
      • Advantages and disadvantages