GPIO writes are going through cache with WRITE_BACK

[dherrendoerfer]

Hi all,

I'm currently trying to get a realtime-simulation working through the IO port of a Raspberry Pi Zero 2 W and I noticed that all my timing was WAY off.
The way things are setup right now is that every write to the GPIO area is going through the MMU and the writes are cached.
Setting up a memory barrier does not really help, because it just waits for the commit and it's slow .
Reading the same IO address back is so far my best solution, but it's super slow.
I literally need to rely on a pin going high to start my internal timing, and >80ns random delay really messes stuff up ;-)
I've modified the /lib/pagetable.cpp to have all sections use write_through, and it's helped a lot.
Is it possible to set up the IO space pagetable entry to have it's own section without cache ? (or write_through)

Love the project btw.!,
Cheers, Dirk

[rsta2]

Hi Dirk,

I don't understand this, because I suppose, the I/O memory space is already set to uncached "shared device" mode:

https://github.com/rsta2/circle/blob/master/lib/pagetable.cpp#L77
https://github.com/rsta2/circle/blob/master/include/circle/armv6mmu.h#L80

Can you please post your own modifications to /lib/pagetable.cpp?

Thanks,
Rene

[dherrendoerfer]

Sure.

diff --git a/lib/pagetable.cpp b/lib/pagetable.cpp
index 42d4431..20f3d8b 100644
--- a/lib/pagetable.cpp
+++ b/lib/pagetable.cpp
@@ -40,8 +40,8 @@
 #elif RASPPI <= 3
 #define MEM_TOTAL_END  ARM_LOCAL_END
 
-#define TTBR_MODE      (  ARM_TTBR_INNER_WRITE_BACK            \
-                        | ARM_TTBR_OUTER_WRITE_BACK)
+#define TTBR_MODE      (  ARM_TTBR_INNER_WRITE_THROUGH         \
+                        | ARM_TTBR_OUTER_WRITE_THROUGH)
 #else
 #define MEM_TOTAL_END  ARM_GIC_END
 #endif

Cheers,
Dirk

[rsta2]

The setting, you have modified, depends on the memory attributes of the memory space, where the page (aka translation) table is located in memory. It is used for the page table walks. This memory is definitely write-back, so the setting should be OK in Circle. It may affect the timing of your application, when you modify this setting, but in an unknown way. I cannot see a reason, why this setting should be modified in Circle.

I have to say, that your timing requirements (80ns resolution) are near to the limits, which are possible with the GPIO hardware of the Raspberry Pi. I read about Circle-based applications, which are able to handle this, but they spent a lot of time to get it working. I suppose, you have to use the class CCPUThrottle to speed up the RPi Zero 2 W to maximum performance. There may also be warm-up effects with the CPU instruction-cache, when the time-critical code is executed for the first time.

[dherrendoerfer]

Yes,
I may have been chasing an entirely different problem - the Pi Zero 2 is a bcm2710. I think it needs barriers before read and after write - it seems I've been reading wrong data, not missing the times to read them.
I added the barriers back, and it's performing ok.
I've played with CCPUThrottle but opted to just set the frequencies manually in config.txt (same values for min and max)
But it seems like I can drive a 65C02 at 4Mhz perfectly through GPIO - and that's great !

Thanks!
Dirk

[dherrendoerfer]

@rsta2 I want to update this, because I found my bug, and it was sort-of a surprise to me.
The thing throwing my IO reads of are writes to the framebuffer - I don't really understand why this never hits on Linux, but on Circle it seems that all hardware access should be protected and kept apart.
My issue was that core 1 was writing to the FB, while the hardware io loop was reading and writing the GPIOs.
I don't know how to get these synched - right now I'm helping myself by having the IO thread on CPU 2 send an event when it's done with a read/write cycle and the graphics thread waits for an event before writing - this helps tons, but I can't say how efficient it is.

[dherrendoerfer]

I believe the same is true for BCM2837 - at least the part where it mentions using barriers to only access one device at a time. At least the mention is still present in the BCM2837 datasheet.
I can get a bad6502 board to you if you want to try yourself.

[rsta2]

At least the mention is still present in the BCM2837 datasheet.

In which BCM2837 datasheet? Can you give me a pointer? I thought, there isn't a specific datasheet for the BCM2837.

[dherrendoerfer]

I have a BCM2837 datasheet (only the first 2 pages are 2837, the rest is 2835)
I modified my code to switch between reading gpio and writing the framebuffer if needed. It's much more stable.

[rsta2]

Probably not an official datasheet? What makes me suspicious is, that Linux doesn't seem to apply these precautions. At least I haven't found them there. Furthermore, it seems impossible to me to apply them in a multi-core environment, were device drivers could access different devices from different cores at the same time. Because of this I do not think, that they're still required. I also did tests on the Raspberry Pi 1 and 2 long ago, and was able to proof this problem on the RPi 1, but not on the RPi 2, which was the first RPi with multiple cores.

[dherrendoerfer]

That's the odd thing: In Linux, this works without serializing the accesses between gpio and FB. I can just swap the sd cards and have one thing work under Linux slowly, or fast and fragile in circle. And the behavior is the same on Zero2 and model 3b+.
I've been through the linux vc4 and gpio code and I see gem doing some ordering, but nothing as far as serializing the accesses.
Like I said, you're free to have a dev board, we don't seem do be located far apart.

[rsta2]

Like I said, you're free to have a dev board, we don't seem do be located far apart.

Thanks, but I think, this is not necessary. I did some tests now on a Raspberry Pi Zero 2 W with this class, which is derived from CMultiCoreSupport:

CIOTest::CIOTest (CScreenDevice *pScreen, CMemorySystem *pMemorySystem)
:	CMultiCoreSupport (pMemorySystem),
	m_pScreen (pScreen),
	m_OutPin (17, GPIOModeOutput),		// pins 17 and 18 are connected
	m_InPin (18, GPIOModeInputPullUp),
	m_nValue (LOW),
	m_nLastTicks (0)
{
}

void CIOTest::Run (unsigned nCore)
{
	switch (nCore)
	{
	case 1:
		for (unsigned i = 0; 1; i++)
		{
			m_pScreen->Rotor (0, i);
		}
		break;

	case 2:
		while (1)
		{
			m_nValue = !m_nValue;

			m_OutPin.Write (m_nValue);

			CTimer::Get ()->nsDelay (200);

			if (m_InPin.Read () != m_nValue)
			{
				LOGWARN ("Out != In");
			}
		}
		break;

	case 3:
		while (1)
		{
			unsigned nTicks = read32 (ARM_SYSTIMER_CLO);

			if (   m_nLastTicks != 0
			    && nTicks - m_nLastTicks > 10)
			{
				LOGWARN ("Invalid ticks (%u / %u)", nTicks, m_nLastTicks);
			}

			m_nLastTicks = nTicks;
		}
	}
}

Core 1 turns the rotor. Core 2 continuously writes an inverted value to pin 17, which is connected to pin 18 on the 40-pin header. After a delay pin 18 is read and compared with the written value. If the values differ, a warning is shown (which never happens).

The delay is necessary, because the CPU cores both use the bus and compete for bus time. It takes some time, to write the current rotor character through to the SDRAM, so that the VC4 can read the pixel values to write them out through the HDMI port.

Core 3 continuously reads a register from another I/O device (the system timer), which is a micro-seconds up-counter. The warning is never shown, so there is no interference with the GPIO peripheral.

To me everything seems to be OK here. The delay is not nice, but as I told, time critical operations should not overlap with framebuffer accesses.

Remains the question, why Linux shows a different behavior here? I can only guess, this may be caused by the newer KMS architecture, where the main CPU manages the HDMI display, not the VC4. The framebuffer memory region could have a write back cache policy in this case and framebuffer writes do not need to go through to the SDRAM. So there is no competition for bus time with the GPIO peripheral.