blob: 03703afc4d302e7eeb7fb4031d494ab750233194
1 | |
2 | Using physical DMA provided by OHCI-1394 FireWire controllers for debugging |
3 | --------------------------------------------------------------------------- |
4 | |
5 | Introduction |
6 | ------------ |
7 | |
8 | Basically all FireWire controllers which are in use today are compliant |
9 | to the OHCI-1394 specification which defines the controller to be a PCI |
10 | bus master which uses DMA to offload data transfers from the CPU and has |
11 | a "Physical Response Unit" which executes specific requests by employing |
12 | PCI-Bus master DMA after applying filters defined by the OHCI-1394 driver. |
13 | |
14 | Once properly configured, remote machines can send these requests to |
15 | ask the OHCI-1394 controller to perform read and write requests on |
16 | physical system memory and, for read requests, send the result of |
17 | the physical memory read back to the requester. |
18 | |
19 | With that, it is possible to debug issues by reading interesting memory |
20 | locations such as buffers like the printk buffer or the process table. |
21 | |
22 | Retrieving a full system memory dump is also possible over the FireWire, |
23 | using data transfer rates in the order of 10MB/s or more. |
24 | |
25 | With most FireWire controllers, memory access is limited to the low 4 GB |
26 | of physical address space. This can be a problem on IA64 machines where |
27 | memory is located mostly above that limit, but it is rarely a problem on |
28 | more common hardware such as x86, x86-64 and PowerPC. |
29 | |
30 | At least LSI FW643e and FW643e2 controllers are known to support access to |
31 | physical addresses above 4 GB, but this feature is currently not enabled by |
32 | Linux. |
33 | |
34 | Together with a early initialization of the OHCI-1394 controller for debugging, |
35 | this facility proved most useful for examining long debugs logs in the printk |
36 | buffer on to debug early boot problems in areas like ACPI where the system |
37 | fails to boot and other means for debugging (serial port) are either not |
38 | available (notebooks) or too slow for extensive debug information (like ACPI). |
39 | |
40 | Drivers |
41 | ------- |
42 | |
43 | The firewire-ohci driver in drivers/firewire uses filtered physical |
44 | DMA by default, which is more secure but not suitable for remote debugging. |
45 | Pass the remote_dma=1 parameter to the driver to get unfiltered physical DMA. |
46 | |
47 | Because the firewire-ohci driver depends on the PCI enumeration to be |
48 | completed, an initialization routine which runs pretty early has been |
49 | implemented for x86. This routine runs long before console_init() can be |
50 | called, i.e. before the printk buffer appears on the console. |
51 | |
52 | To activate it, enable CONFIG_PROVIDE_OHCI1394_DMA_INIT (Kernel hacking menu: |
53 | Remote debugging over FireWire early on boot) and pass the parameter |
54 | "ohci1394_dma=early" to the recompiled kernel on boot. |
55 | |
56 | Tools |
57 | ----- |
58 | |
59 | firescope - Originally developed by Benjamin Herrenschmidt, Andi Kleen ported |
60 | it from PowerPC to x86 and x86_64 and added functionality, firescope can now |
61 | be used to view the printk buffer of a remote machine, even with live update. |
62 | |
63 | Bernhard Kaindl enhanced firescope to support accessing 64-bit machines |
64 | from 32-bit firescope and vice versa: |
65 | - http://v3.sk/~lkundrak/firescope/ |
66 | |
67 | and he implemented fast system dump (alpha version - read README.txt): |
68 | - http://halobates.de/firewire/firedump-0.1.tar.bz2 |
69 | |
70 | There is also a gdb proxy for firewire which allows to use gdb to access |
71 | data which can be referenced from symbols found by gdb in vmlinux: |
72 | - http://halobates.de/firewire/fireproxy-0.33.tar.bz2 |
73 | |
74 | The latest version of this gdb proxy (fireproxy-0.34) can communicate (not |
75 | yet stable) with kgdb over an memory-based communication module (kgdbom). |
76 | |
77 | Getting Started |
78 | --------------- |
79 | |
80 | The OHCI-1394 specification regulates that the OHCI-1394 controller must |
81 | disable all physical DMA on each bus reset. |
82 | |
83 | This means that if you want to debug an issue in a system state where |
84 | interrupts are disabled and where no polling of the OHCI-1394 controller |
85 | for bus resets takes place, you have to establish any FireWire cable |
86 | connections and fully initialize all FireWire hardware __before__ the |
87 | system enters such state. |
88 | |
89 | Step-by-step instructions for using firescope with early OHCI initialization: |
90 | |
91 | 1) Verify that your hardware is supported: |
92 | |
93 | Load the firewire-ohci module and check your kernel logs. |
94 | You should see a line similar to |
95 | |
96 | firewire_ohci 0000:15:00.1: added OHCI v1.0 device as card 2, 4 IR + 4 IT |
97 | ... contexts, quirks 0x11 |
98 | |
99 | when loading the driver. If you have no supported controller, many PCI, |
100 | CardBus and even some Express cards which are fully compliant to OHCI-1394 |
101 | specification are available. If it requires no driver for Windows operating |
102 | systems, it most likely is. Only specialized shops have cards which are not |
103 | compliant, they are based on TI PCILynx chips and require drivers for Win- |
104 | dows operating systems. |
105 | |
106 | The mentioned kernel log message contains the string "physUB" if the |
107 | controller implements a writable Physical Upper Bound register. This is |
108 | required for physical DMA above 4 GB (but not utilized by Linux yet). |
109 | |
110 | 2) Establish a working FireWire cable connection: |
111 | |
112 | Any FireWire cable, as long at it provides electrically and mechanically |
113 | stable connection and has matching connectors (there are small 4-pin and |
114 | large 6-pin FireWire ports) will do. |
115 | |
116 | If an driver is running on both machines you should see a line like |
117 | |
118 | firewire_core 0000:15:00.1: created device fw1: GUID 00061b0020105917, S400 |
119 | |
120 | on both machines in the kernel log when the cable is plugged in |
121 | and connects the two machines. |
122 | |
123 | 3) Test physical DMA using firescope: |
124 | |
125 | On the debug host, make sure that /dev/fw* is accessible, |
126 | then start firescope: |
127 | |
128 | $ firescope |
129 | Port 0 (/dev/fw1) opened, 2 nodes detected |
130 | |
131 | FireScope |
132 | --------- |
133 | Target : <unspecified> |
134 | Gen : 1 |
135 | [Ctrl-T] choose target |
136 | [Ctrl-H] this menu |
137 | [Ctrl-Q] quit |
138 | |
139 | ------> Press Ctrl-T now, the output should be similar to: |
140 | |
141 | 2 nodes available, local node is: 0 |
142 | 0: ffc0, uuid: 00000000 00000000 [LOCAL] |
143 | 1: ffc1, uuid: 00279000 ba4bb801 |
144 | |
145 | Besides the [LOCAL] node, it must show another node without error message. |
146 | |
147 | 4) Prepare for debugging with early OHCI-1394 initialization: |
148 | |
149 | 4.1) Kernel compilation and installation on debug target |
150 | |
151 | Compile the kernel to be debugged with CONFIG_PROVIDE_OHCI1394_DMA_INIT |
152 | (Kernel hacking: Provide code for enabling DMA over FireWire early on boot) |
153 | enabled and install it on the machine to be debugged (debug target). |
154 | |
155 | 4.2) Transfer the System.map of the debugged kernel to the debug host |
156 | |
157 | Copy the System.map of the kernel be debugged to the debug host (the host |
158 | which is connected to the debugged machine over the FireWire cable). |
159 | |
160 | 5) Retrieving the printk buffer contents: |
161 | |
162 | With the FireWire cable connected, the OHCI-1394 driver on the debugging |
163 | host loaded, reboot the debugged machine, booting the kernel which has |
164 | CONFIG_PROVIDE_OHCI1394_DMA_INIT enabled, with the option ohci1394_dma=early. |
165 | |
166 | Then, on the debugging host, run firescope, for example by using -A: |
167 | |
168 | firescope -A System.map-of-debug-target-kernel |
169 | |
170 | Note: -A automatically attaches to the first non-local node. It only works |
171 | reliably if only connected two machines are connected using FireWire. |
172 | |
173 | After having attached to the debug target, press Ctrl-D to view the |
174 | complete printk buffer or Ctrl-U to enter auto update mode and get an |
175 | updated live view of recent kernel messages logged on the debug target. |
176 | |
177 | Call "firescope -h" to get more information on firescope's options. |
178 | |
179 | Notes |
180 | ----- |
181 | Documentation and specifications: http://halobates.de/firewire/ |
182 | |
183 | FireWire is a trademark of Apple Inc. - for more information please refer to: |
184 | https://en.wikipedia.org/wiki/FireWire |
185 |