PCIe Protocol
PCIe is a high-speed serial computer expansion bus standard. PCIe was designed as a high-speed replacement for the PCI and AGP standards. The data transmitted is sent over lanes in both directions at the same time, each lane is capable of transfer speeds of around 250 MB/s and each slot can be scaled from 1 to 32 lanes. With 16 lanes, PCIe can support a bandwidth of up to 4 GB/s.
PCIe works like a network, with each card connected to a network switch through a dedicated set of wires. Each card has its own physical connection to the switch fabric, and the communication takes the form of packets transmitted over these dedicated lines, with flow control, error detection, and retransmissions. There are no MAC addresses, but the card’s physical position instead to define it, before it’s allocated with high-level means of addressing it.
Simple Bus Transaction
The CPU itself performs a simple write operation on its own bus which has a direct connection to the PCIe bus. So what happens is that the chipset (which, in PCIe terms functions as a Root Complex) generates a Memory Write packet for transmission over the bus. This packet consists of a header, which is either 3 or 4 32-bit words long (depending on if 32 or 64 bit addressing is used) and one 32-bit word containing the word to be written.
This packet is then transmitted on the chipset’s PCIe port. The target peripheral may be connected directly to the chipset, or there may be a switch network between them. The packet is routed to the peripheral, decoded, and executed by performing the desired write operation.
32bit vs 64bit
For Addresses below 4 GB, Requesters must use the 32-bit format. The behavior of the receiver is not specified if a 64-bit format request addressing below 4 GB (i.e., with the upper 32 bits of address all 0) is received.
Routing
There are three routing methods: By address, by ID, and implicit. By address, routing is applied for memory and I/O Requests (read and write). Implicit routing is used only for certain message TLPs (Transfer Layer Package), such as broadcasts from Root Complex and messages that always go to the Root Complex. All other TLPs are routed by ID.
PCIe Bus Mastering
This allows the peripheral to access the CPU’s memory directly (DMA) or exchange TLPs with peer peripherals (to the extent that the switching entities support that). Well, there are two things that need to happen first, as with any PCI device: The peripheral needs to be granted bus mastering by setting the “Bus Master Enable” bit in one of the standard configuration registers. The second thing is that the driver software needs to inform the peripheral about the relevant buffer’s physical address, by writing to a BAR-mapped register.
Interrupts
INTx interrupts are supported for the sake of compatibility with legacy software, and also in order to allow bridging between classic PCI buses and PCIe. Since INTx interrupts are level triggered, there’s a TLP packet for saying that the line has been asserted, and another that it has been deasserted.
MSI was introduced in (conventional) PCI 2.2. PCIe does exactly the same to generate an MSI: Signaling an interrupt merely consists of sending a TLP over the bus, which is simply a posted Write Request, with a special address, which the host has written into the peripheral’s configuration space during initialization. Linux can then call the correct interrupt routine, without the need to guess who generated the interrupt. Neither is it really necessary to “clear” the interrupt if the peripheral doesn’t need the acknowledgment.
Further Reading
http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2
http://xillybus.com/tutorials/pci-express-dma-requests-completions
http://www.fpga.world/_altera/html/ref/40nm_workshops/PCIExpress_p.pdf
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
