CPU Thermal throttling

Feature: https://source.puri.sm/Librem5/use-cases/issues/75

We don't want the phone to be unusably hot (and breaking the first law of thermodynamics is not allowed) so this issue is to track the efforts to try and control the heat produced by the phone.

added feature label

removed feature label

changed title from Thermal management to Thermal throttling

changed the description

What does this mean exactly? What do we want to do? apply some policy to the linux thermal cooling devices?

From the kernel perspective, we already include the available cooling devices in the thermal subsystem:

purism@pureos:/sys/class/thermal$ cat cooling_device*/type
38000000.gpu
thermal-cpufreq-0
purism@pureos:/sys/class/thermal$ cat cooling_device*/max_state
6
2
purism@pureos:/sys/class/thermal$ cat cooling_device*/cur_state
0
0

• Is "cpuidle" missing?
• I don't know what application would typically be in charge to control this.

so the goal is for the user to be able to stress all cpus, and we still put them in sleep states in case the temperature rises over a given threshold?

@guido.gunther @angus.ainslie in case you have any experience with that already, I'm happy for hints before I look into this - if only knowing what parts of the stack are involved with this. thanks.

I don't know what the user interface is supposed to be here yet.

On our 5.2 kernel we only have

ls -l /sys/devices/virtual/thermal/
total 0
drwxr-xr-x 3 root root 0 Jan  1  1970 cooling_device0
drwxr-xr-x 3 root root 0 Jan  1  1970 thermal_zone0
drwxr-xr-x 3 root root 0 Jan  1  1970 thermal_zone1
drwxr-xr-x 3 root root 0 Jan  1  1970 thermal_zone2

with the cooling_device0 being the "gpu".

And, regardless of "missing" cooling devices, QORIQ_THERMAL (our tmu driver) says it uses "cpufreq" (only. not cpuidle).

also, (when looking at powercap?), adding

--- a/arch/arm64/configs/librem5-devkit_defconfig
+++ b/arch/arm64/configs/librem5-devkit_defconfig
@@ -562,6 +562,7 @@ CONFIG_PHY_XGENE=y
 CONFIG_PHY_FSL_IMX8MQ_USB=m
 CONFIG_PHY_MIXEL_MIPI_DPHY=y
 CONFIG_POWERCAP=y
+CONFIG_IDLE_INJECT=y
 CONFIG_NVMEM_IMX_OCOTP=y
 CONFIG_VALIDATE_FS_PARSER=y
 CONFIG_EXT2_FS=y

doesn't make a difference.

in our next-20190607 tree we've had

purism@pureos:/sys/class/thermal$ ls
cooling_device0  cooling_device1  thermal_zone0  thermal_zone1  thermal_zone2
purism@pureos:/sys/class/thermal$ cat cooling_device*/type
38000000.gpu
thermal-cpufreq-0

where thermal-cpufreq-%x should be registered by IMX_THERMAL (which we have configured in).

but despite that cooling driver (using cpufreq) being missing in our 5.2 tree, there currently is no cooling driver that uses cpuidle. The last effort to add that would be https://lore.kernel.org/linux-pm/1522945005-7165-1-git-send-email-daniel.lezcano@linaro.org/

So currently open review question: why don't we have the cpufreq cooling device on 5.2? That should be fixed anyways, even if it won't have much impact.

would make sense to debug this while moving to 5.3-rc1 I guess...

TODO:

• test / debug and bring back thermal-cpufreq on 5.3-rc1 / next
• test / write https://lore.kernel.org/linux-pm/1522945005-7165-1-git-send-email-daniel.lezcano@linaro.org/
• what other IPs are able to have thermal drivers?

• cpufreq itself and thermal-cpufreq is back: https://source.puri.sm/Librem5/linux-next/merge_requests/11

---

---

• test a thermal-idle driver for cpuidle based on https://lore.kernel.org/linux-pm/1522945005-7165-1-git-send-email-daniel.lezcano@linaro.org/

MR: https://source.puri.sm/Librem5/linux-next/merge_requests/12

During all following tests, we run stress -c 4 while watching the following files. That's how we do it:

#!/bin/bash
while true
do
	cat /sys/class/thermal/thermal_zone0/temp
	cat /sys/class/thermal/cooling_device0/cur_state

	printf "%s %s %s" "$(awk '{print $1}' /proc/uptime)" \
		"$(cat /sys/class/thermal/thermal_zone0/temp)" \
		"$(cat /sys/class/thermal/cooling_device0/cur_state)" >> result
	printf "\n" >> result
	sleep 10
done

1st Test without these changes: Run stress -c 4 and verify that the CPUs heat up to 90°C and the system shuts down. That's the easy part.

---

Test with 60 °C target temperature, in order to hit it earlier than 80, for testing:

--- a/arch/arm64/boot/dts/freescale/imx8mq.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx8mq.dtsi
@@ -221,7 +221,7 @@
 
                        trips {
                                cpu_alert: cpu-alert {
-                                       temperature = <80000>;
+                                       temperature = <60000>;
                                        hysteresis = <2000>;
                                        type = "passive";
                                };

Note: "cooling_device0" is identifiable by having type read thermal-idle-0.

Result:

Behaviour is as expected at my place (without a fan). 60 °C are hard to keep.

Comparison result: The same code, but unmodified devicetree description: 80°C target termperature:

So 80°C are of course easier to keep than 60°C. Remember, stress -c 4 is always running during these tests.

TL;DR: thermal-idle works as an out-of-tree solution and is indeed effective in cooling under load. 80°C should be able to keep under a "load average" of 4, i.e. full CPU load.

additional candidate for thermal cooling driver: devfreq_cooling which depends on https://source.puri.sm/Librem5/linux-next/issues/17

@martin.kepplinger during these tests that you used to produce these plots, what was the ambient temperature? Did you have your heatsink facing down towards a table or pointing upwards towards the ceiling/sky or at some angle?

@eric.kuzmenko I'm not exactly sure, but ambient temp has been circa 23°C. The devkit was lying display-down, heatsink-up on my desk.

@martin.kepplinger when you run these sorts of thermal tests please keep the heatsink facing down towards your table with the display facing up. Since the phone is going to have the SoC completely enclosed (with some heatsinking inside), there won't be convection with the environment like you can get with the heatsink fins on the dev kit. The best way to reproduce such a scenario with the dev kit is by making the heatsink point downward (it will get hotter because of this).

Also, please record your ambient temperature when you run thermal tests.

Thanks for the tip, Eric. The purpose of this test was to show that the driver works; at least for that it's been good enough.

I did a test run with the current imx8-linux-next-dev branch based on 5.3-rc2: Ambient temp: 23,5 °C. devkit display-up facing. stress -c 4 heats it up to 80°C. The thermal driver kicks in and does idle-injection on the cpu. The temperature never goes beyond 82 °C..

Note: This is in no way a situation we want the final phone to be in! The CPU makes up for everything else pulling way too much power (see the open issues), and gets up to 99% slower in the process. It only shows that the thermal driver works. This test (stress -c 4) currently makes the phone (lockscreen) unusably slow. A few minutes after stopping "stress -c 4", the cpu still won't be able to cool down and stays at 80°C + 99% idle-injection. Indeed this is important, but right now, results in really bad user experience.

thermal-idle upstream bugreport + followup / call for update: https://lore.kernel.org/linux-pm/20190805051111.24318-1-martin.kepplinger@puri.sm/

The thermal-idle driver breaks S3 suspend :( https://lore.kernel.org/linux-pm/02ec23c3-37ee-4e9f-56a4-453a30a29747@puri.sm/T/#meb0ae9702461e5ce3e49da0fa221bc9a077769a0

(resume doesn't seem to work anyways)

changed title from Thermal throttling to CPU Thermal throttling

changed the description

We'll close the issue when we get a (suspend-) fixed version from the author.

added critical label

added 1 label

related bug: https://source.puri.sm/Librem5/linux-next/issues/81

mentioned in issue #81 (closed)

we have a separate issue for this breaking suspend https://source.puri.sm/Librem5/linux-next/issues/84 so closing this one.

closed

mentioned in issue #17