diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 27792dbee10922a19b78e74b6fa3a03368f505b9..c177c65c3b4120d6320bb020e8ca28283ee0e454 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6457,3 +6457,11 @@ memory, and other data can't be written using xmon commands. off xmon is disabled. + + pbha= [ARM64] + Format: { enable | user } + Enabled PBHA bit0. + enable kernel and user will update PBHA bit0 for their + pte entry. + user only select user task will update PBHA bit0 for + their pte entry. diff --git a/Documentation/arm64/index.rst b/Documentation/arm64/index.rst index 937634c499798bcd22a30764c474a934681c02a0..b65de0107c977773e61cfd74bbdfe8beb815c5c1 100644 --- a/Documentation/arm64/index.rst +++ b/Documentation/arm64/index.rst @@ -17,6 +17,7 @@ ARM64 Architecture legacy_instructions memory memory-tagging-extension + pbha perf pointer-authentication silicon-errata diff --git a/Documentation/arm64/pbha.rst b/Documentation/arm64/pbha.rst new file mode 100644 index 0000000000000000000000000000000000000000..d8a3df8982a5e75ea4a7ce6878e756fc8d3f6b5e --- /dev/null +++ b/Documentation/arm64/pbha.rst @@ -0,0 +1,85 @@ +======================================================= +Page Based Hardware Attribute support for AArch64 Linux +======================================================= + +Page Based Hardware Attributes (PBHA) allow the OS to trigger IMPLEMENTATION +DEFINED behaviour associated with a memory access. For example, this may be +taken as a hint to a System Cache whether it should cache the location that +has been accessed. + +PBHA consists of four bits in the leaf page table entries for a virtual +address, that are sent with any memory access via that virtual address. + +IMPLEMENTATION DEFINED behaviour is not specified by the arm-arm, meaning +it varies between SoCs. There may be unexpected side effects when PBHA +bits are used or combined. +For example, a PBHA bit may be taken as a hint to the Memory Controller that +it should encrypt/decrypt the memory in DRAM. If the CPU has multiple virtual +aliases of the address, accesses that are made without this PBHA bit set may +cause corruption. + + +Use by virtual machines using KVM +--------------------------------- + +KVM allows an OS in a virtual machine to configure its own page tables. A +virtual machine can also configure PBHA bits in its page tables. To prevent +side effects that could affect the hypervisor, KVM will only allow +combinations of PBHA bits that only affect performance. Values that cause +changes to the data are forbidden as the Hypervisor and VMM have aliases of +the guest memory, and may swap it to/from disk. + +The list of bits to allow is built from the firmware list of PBHA bit +combinations that only affect performance. Because the guest can choose +not to set all the bits in a value, (e.g. allowing 5 implicitly allows 1 +and 4), the values supported may differ between a host and guest. + +PBHA is only supported for a guest if KVM supports the mechanism the CPU uses +to combine the values from stage1 and stage2 translation. The mechanism is not +advertised, so which mechanism each CPU uses must also be known by the kernel. + + +Use by device drivers +--------------------- + +Device drivers should discover the PBHA value to use for a mapping from the +device's firmware description as these will vary between SoCs. If the value +is also listed by firmware as only affecting performance, it can be added to +the pgprot with pgprot_pbha(). + +Values that require all other aliases to be removed are not supported. + + +Linux's expectations around PBHA +-------------------------------- + +'IMPLEMENTATION DEFINED' describes a huge range of possible behaviours. +Linux expects PBHA to behave in the same way as the read/write allocate hints +for a memory type. Below is an incomplete list of expectations: + + * PBHA values have the same meaning for all CPUs in the SoC. + * Use of the PBHA value does not cause mismatched type, shareability or + cacheability, it does not take precedence over the stage2 attributes, or + HCR_EL2 controls. + * If a PBHA value requires all other aliases to be removed, higher exception + levels do not have a concurrent alias. (This includes Secure World). + * Break before make is sufficient when changing the PBHA value. + * PBHA values used by a page can be changed independently without further side + effects. + * Save/restoring the page contents via a PBHA=0 mapping does not corrupt the + values once a non-zero PBHA mapping is re-created. + * The hypervisor may clean+invalidate to the PoC via a PBHA=0 mapping prior to + save/restore to cleanup mismatched attributes. This does not corrupt the + values after save/restore once a non-zero PBHA mapping is re-created. + * Cache maintenance via a PBHA=0 mapping to prevent stale data being visible + when mismatched attributes occur is sufficient even if the subsequent + mapping has a non-zero PBHA value. + * The OS/hypervisor can clean-up a page by removing all non-zero PBHA mappings, + then writing new data via PBHA=0 mapping of the same type, shareability and + cacheability. After this, only the new data is visible for data accesses. + * For instruction-fetch, the same maintenance as would be performed against a + PBHA=0 page is sufficient. (which with DIC+IDC, may be none at all). + * The behaviour enabled by PBHA should not depend on the size of the access, or + whether other SoC hardware under the control of the OS is enabled and + configured. + * EL2 is able to at least force stage1 PBHA bits to zero. diff --git a/Documentation/devicetree/bindings/arm/cpu.yaml b/Documentation/devicetree/bindings/arm/cpu.yaml new file mode 100644 index 0000000000000000000000000000000000000000..8ae27f370a1250a2731b54394f9e932612c240bc --- /dev/null +++ b/Documentation/devicetree/bindings/arm/cpu.yaml @@ -0,0 +1,537 @@ +# SPDX-License-Identifier: GPL-2.0 +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/arm/cpu.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: ARM CPUs bindings + +maintainers: + - Lorenzo Pieralisi + +description: |+ + The device tree allows to describe the layout of CPUs in a system through + the "cpus" node, which in turn contains a number of subnodes (ie "cpu") + defining properties for every cpu. + + Bindings for CPU nodes follow the Devicetree Specification, available from: + + https://www.devicetree.org/specifications/ + + with updates for 32-bit and 64-bit ARM systems provided in this document. + + ================================ + Convention used in this document + ================================ + + This document follows the conventions described in the Devicetree + Specification, with the addition: + + - square brackets define bitfields, eg reg[7:0] value of the bitfield in + the reg property contained in bits 7 down to 0 + + ===================================== + cpus and cpu node bindings definition + ===================================== + + The ARM architecture, in accordance with the Devicetree Specification, + requires the cpus and cpu nodes to be present and contain the properties + described below. + +properties: + reg: + maxItems: 1 + description: | + Usage and definition depend on ARM architecture version and + configuration: + + On uniprocessor ARM architectures previous to v7 + this property is required and must be set to 0. + + On ARM 11 MPcore based systems this property is + required and matches the CPUID[11:0] register bits. + + Bits [11:0] in the reg cell must be set to + bits [11:0] in CPU ID register. + + All other bits in the reg cell must be set to 0. + + On 32-bit ARM v7 or later systems this property is + required and matches the CPU MPIDR[23:0] register + bits. + + Bits [23:0] in the reg cell must be set to + bits [23:0] in MPIDR. + + All other bits in the reg cell must be set to 0. + + On ARM v8 64-bit systems this property is required + and matches the MPIDR_EL1 register affinity bits. + + * If cpus node's #address-cells property is set to 2 + + The first reg cell bits [7:0] must be set to + bits [39:32] of MPIDR_EL1. + + The second reg cell bits [23:0] must be set to + bits [23:0] of MPIDR_EL1. + + * If cpus node's #address-cells property is set to 1 + + The reg cell bits [23:0] must be set to bits [23:0] + of MPIDR_EL1. + + All other bits in the reg cells must be set to 0. + + compatible: + enum: + - arm,arm710t + - arm,arm720t + - arm,arm740t + - arm,arm7ej-s + - arm,arm7tdmi + - arm,arm7tdmi-s + - arm,arm9es + - arm,arm9ej-s + - arm,arm920t + - arm,arm922t + - arm,arm925 + - arm,arm926e-s + - arm,arm926ej-s + - arm,arm940t + - arm,arm946e-s + - arm,arm966e-s + - arm,arm968e-s + - arm,arm9tdmi + - arm,arm1020e + - arm,arm1020t + - arm,arm1022e + - arm,arm1026ej-s + - arm,arm1136j-s + - arm,arm1136jf-s + - arm,arm1156t2-s + - arm,arm1156t2f-s + - arm,arm1176jzf + - arm,arm1176jz-s + - arm,arm1176jzf-s + - arm,arm11mpcore + - arm,armv8 # Only for s/w models + - arm,cortex-a5 + - arm,cortex-a7 + - arm,cortex-a8 + - arm,cortex-a9 + - arm,cortex-a12 + - arm,cortex-a15 + - arm,cortex-a17 + - arm,cortex-a32 + - arm,cortex-a34 + - arm,cortex-a35 + - arm,cortex-a53 + - arm,cortex-a55 + - arm,cortex-a57 + - arm,cortex-a65 + - arm,cortex-a72 + - arm,cortex-a73 + - arm,cortex-a75 + - arm,cortex-a76 + - arm,cortex-a77 + - arm,cortex-m0 + - arm,cortex-m0+ + - arm,cortex-m1 + - arm,cortex-m3 + - arm,cortex-m4 + - arm,cortex-r4 + - arm,cortex-r5 + - arm,cortex-r7 + - arm,neoverse-e1 + - arm,neoverse-n1 + - brcm,brahma-b15 + - brcm,brahma-b53 + - brcm,vulcan + - cavium,thunder + - cavium,thunder2 + - faraday,fa526 + - intel,sa110 + - intel,sa1100 + - marvell,feroceon + - marvell,mohawk + - marvell,pj4a + - marvell,pj4b + - marvell,sheeva-v5 + - marvell,sheeva-v7 + - nvidia,tegra132-denver + - nvidia,tegra186-denver + - nvidia,tegra194-carmel + - qcom,krait + - qcom,kryo + - qcom,kryo260 + - qcom,kryo280 + - qcom,kryo385 + - qcom,kryo468 + - qcom,kryo485 + - qcom,scorpion + + enable-method: + $ref: '/schemas/types.yaml#/definitions/string' + oneOf: + # On ARM v8 64-bit this property is required + - enum: + - psci + - spin-table + # On ARM 32-bit systems this property is optional + - enum: + - actions,s500-smp + - allwinner,sun6i-a31 + - allwinner,sun8i-a23 + - allwinner,sun9i-a80-smp + - allwinner,sun8i-a83t-smp + - amlogic,meson8-smp + - amlogic,meson8b-smp + - arm,realview-smp + - aspeed,ast2600-smp + - brcm,bcm11351-cpu-method + - brcm,bcm23550 + - brcm,bcm2836-smp + - brcm,bcm63138 + - brcm,bcm-nsp-smp + - brcm,brahma-b15 + - marvell,armada-375-smp + - marvell,armada-380-smp + - marvell,armada-390-smp + - marvell,armada-xp-smp + - marvell,98dx3236-smp + - marvell,mmp3-smp + - mediatek,mt6589-smp + - mediatek,mt81xx-tz-smp + - qcom,gcc-msm8660 + - qcom,kpss-acc-v1 + - qcom,kpss-acc-v2 + - renesas,apmu + - renesas,r9a06g032-smp + - rockchip,rk3036-smp + - rockchip,rk3066-smp + - socionext,milbeaut-m10v-smp + - ste,dbx500-smp + - ti,am3352 + - ti,am4372 + + cpu-release-addr: + $ref: '/schemas/types.yaml#/definitions/uint64' + + description: + Required for systems that have an "enable-method" + property value of "spin-table". + On ARM v8 64-bit systems must be a two cell + property identifying a 64-bit zero-initialised + memory location. + + cpu-idle-states: + $ref: '/schemas/types.yaml#/definitions/phandle-array' + description: | + List of phandles to idle state nodes supported + by this cpu (see ./idle-states.yaml). + + capacity-dmips-mhz: + $ref: '/schemas/types.yaml#/definitions/uint32' + description: + u32 value representing CPU capacity (see ./cpu-capacity.txt) in + DMIPS/MHz, relative to highest capacity-dmips-mhz + in the system. + + dynamic-power-coefficient: + $ref: '/schemas/types.yaml#/definitions/uint32' + description: + A u32 value that represents the running time dynamic + power coefficient in units of uW/MHz/V^2. The + coefficient can either be calculated from power + measurements or derived by analysis. + + The dynamic power consumption of the CPU is + proportional to the square of the Voltage (V) and + the clock frequency (f). The coefficient is used to + calculate the dynamic power as below - + + Pdyn = dynamic-power-coefficient * V^2 * f + + where voltage is in V, frequency is in MHz. + + power-domains: + $ref: '/schemas/types.yaml#/definitions/phandle-array' + description: + List of phandles and PM domain specifiers, as defined by bindings of the + PM domain provider (see also ../power_domain.txt). + + power-domain-names: + $ref: '/schemas/types.yaml#/definitions/string-array' + description: + A list of power domain name strings sorted in the same order as the + power-domains property. + + For PSCI based platforms, the name corresponding to the index of the PSCI + PM domain provider, must be "psci". + + qcom,saw: + $ref: '/schemas/types.yaml#/definitions/phandle' + description: | + Specifies the SAW* node associated with this CPU. + + Required for systems that have an "enable-method" property + value of "qcom,kpss-acc-v1" or "qcom,kpss-acc-v2" + + * arm/msm/qcom,saw2.txt + + qcom,acc: + $ref: '/schemas/types.yaml#/definitions/phandle' + description: | + Specifies the ACC* node associated with this CPU. + + Required for systems that have an "enable-method" property + value of "qcom,kpss-acc-v1" or "qcom,kpss-acc-v2" + + * arm/msm/qcom,kpss-acc.txt + + rockchip,pmu: + $ref: '/schemas/types.yaml#/definitions/phandle' + description: | + Specifies the syscon node controlling the cpu core power domains. + + Optional for systems that have an "enable-method" + property value of "rockchip,rk3066-smp" + While optional, it is the preferred way to get access to + the cpu-core power-domains. + + secondary-boot-reg: + $ref: '/schemas/types.yaml#/definitions/uint32' + description: | + Required for systems that have an "enable-method" property value of + "brcm,bcm11351-cpu-method", "brcm,bcm23550" or "brcm,bcm-nsp-smp". + + This includes the following SoCs: | + BCM11130, BCM11140, BCM11351, BCM28145, BCM28155, BCM21664, BCM23550 + BCM58522, BCM58525, BCM58535, BCM58622, BCM58623, BCM58625, BCM88312 + + The secondary-boot-reg property is a u32 value that specifies the + physical address of the register used to request the ROM holding pen + code release a secondary CPU. The value written to the register is + formed by encoding the target CPU id into the low bits of the + physical start address it should jump to. + +if: + # If the enable-method property contains one of those values + properties: + enable-method: + contains: + enum: + - brcm,bcm11351-cpu-method + - brcm,bcm23550 + - brcm,bcm-nsp-smp + # and if enable-method is present + required: + - enable-method + +then: + required: + - secondary-boot-reg + +required: + - device_type + - reg + - compatible + +dependencies: + rockchip,pmu: [enable-method] + +additionalProperties: true + +examples: + - | + cpus { + #size-cells = <0>; + #address-cells = <1>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,cortex-a15"; + reg = <0x0>; + }; + + cpu@1 { + device_type = "cpu"; + compatible = "arm,cortex-a15"; + reg = <0x1>; + }; + + cpu@100 { + device_type = "cpu"; + compatible = "arm,cortex-a7"; + reg = <0x100>; + }; + + cpu@101 { + device_type = "cpu"; + compatible = "arm,cortex-a7"; + reg = <0x101>; + }; + }; + + - | + // Example 2 (Cortex-A8 uniprocessor 32-bit system): + cpus { + #size-cells = <0>; + #address-cells = <1>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,cortex-a8"; + reg = <0x0>; + }; + }; + + - | + // Example 3 (ARM 926EJ-S uniprocessor 32-bit system): + cpus { + #size-cells = <0>; + #address-cells = <1>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,arm926ej-s"; + reg = <0x0>; + }; + }; + + - | + // Example 4 (ARM Cortex-A57 64-bit system): + cpus { + #size-cells = <0>; + #address-cells = <2>; + + cpu@0 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x0>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@1 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x1>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x100>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@101 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x101>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@10000 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x10000>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@10001 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x10001>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@10100 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x10100>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@10101 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x0 0x10101>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100000000 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x0>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100000001 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x1>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100000100 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x100>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100000101 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x101>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100010000 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x10000>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100010001 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x10001>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100010100 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x10100>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + + cpu@100010101 { + device_type = "cpu"; + compatible = "arm,cortex-a57"; + reg = <0x1 0x10101>; + enable-method = "spin-table"; + cpu-release-addr = <0 0x20000000>; + }; + }; +... diff --git a/Documentation/devicetree/bindings/arm/cpus.yaml b/Documentation/devicetree/bindings/arm/cpus.yaml index 14cd727d3c4b75c12afed857664d4dc5822a7f51..326e393d4de1bbd22ee2e07c61962044cf031c2d 100644 --- a/Documentation/devicetree/bindings/arm/cpus.yaml +++ b/Documentation/devicetree/bindings/arm/cpus.yaml @@ -1,537 +1,91 @@ -# SPDX-License-Identifier: GPL-2.0 +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) %YAML 1.2 --- -$id: http://devicetree.org/schemas/arm/cpus.yaml# -$schema: http://devicetree.org/meta-schemas/core.yaml# +$id: "http://devicetree.org/schemas/arm/cpus.yaml#" +$schema: "http://devicetree.org/meta-schemas/core.yaml#" -title: ARM CPUs bindings +title: CPUS, a container for CPU subnodes -maintainers: - - Lorenzo Pieralisi - -description: |+ +description: | The device tree allows to describe the layout of CPUs in a system through the "cpus" node, which in turn contains a number of subnodes (ie "cpu") - defining properties for every cpu. - - Bindings for CPU nodes follow the Devicetree Specification, available from: - - https://www.devicetree.org/specifications/ - - with updates for 32-bit and 64-bit ARM systems provided in this document. - - ================================ - Convention used in this document - ================================ - - This document follows the conventions described in the Devicetree - Specification, with the addition: - - - square brackets define bitfields, eg reg[7:0] value of the bitfield in - the reg property contained in bits 7 down to 0 - - ===================================== - cpus and cpu node bindings definition - ===================================== - - The ARM architecture, in accordance with the Devicetree Specification, - requires the cpus and cpu nodes to be present and contain the properties - described below. + defining properties for every CPU. + + Properties of the CPU integration that are common to all CPUs can be described + in the cpus node. + + ARM CPUs with the FEAT_HPDS2 feature allow an IMPLEMENTATION DEFINED + hardware attribute to be encoded in the leaf page table entries and + sent with any transaction that makes an access via that entry. + + Four bits are used in the page-tables. It is likely the individual bits will + be combined and used a a four bit value. The impact of any particular value is + up to the implementation. + + 0 is defined as a 'safe default setting' that behaves as if the feature + were not implemented. Other values may be unsafe, having coherency or + correctness issues leading to data-corruption or deadlock. + + This binding lists the additional values that only have a performance cost + (or benefit), and values that can only be used if all mappings have the same + PBHA value. + For both cases, all affected values should be listed. If setting bit-2 + requires no aliases, then the values 2, 4, 6 etc should be listed. + + A hypervisor can only control individual bits, and may choose to only enable + bits that can only be used to build other performance-only values. + e.g. the value 5 is listed, but enabling bit-0 and bit-2 would allow a guest + to configure the values 1 or 4 too. If these 'decomposed' values only + affected performance, they should also be listed. + + The list does not need to be in numeric order, but a hypervisor may make use + of the order when enabling bits. + + The presence of a 'arm,pbha-no-aliases' property indicates that higher + exception levels and secure-world firmware do not have a mapping of any memory + in the memory node or UEFI memory map, other than those with a reserved-memory + entry or EFIReserved memory attribute. + Firmware mappings created based on requests from the normal world do not use + any of the arm,pbha-no-aliases values, or take the PBHA value to use as an + argument. properties: - reg: - maxItems: 1 - description: | - Usage and definition depend on ARM architecture version and - configuration: - - On uniprocessor ARM architectures previous to v7 - this property is required and must be set to 0. - - On ARM 11 MPcore based systems this property is - required and matches the CPUID[11:0] register bits. - - Bits [11:0] in the reg cell must be set to - bits [11:0] in CPU ID register. - - All other bits in the reg cell must be set to 0. - - On 32-bit ARM v7 or later systems this property is - required and matches the CPU MPIDR[23:0] register - bits. - - Bits [23:0] in the reg cell must be set to - bits [23:0] in MPIDR. - - All other bits in the reg cell must be set to 0. - - On ARM v8 64-bit systems this property is required - and matches the MPIDR_EL1 register affinity bits. - - * If cpus node's #address-cells property is set to 2 - - The first reg cell bits [7:0] must be set to - bits [39:32] of MPIDR_EL1. - - The second reg cell bits [23:0] must be set to - bits [23:0] of MPIDR_EL1. - - * If cpus node's #address-cells property is set to 1 - - The reg cell bits [23:0] must be set to bits [23:0] - of MPIDR_EL1. - - All other bits in the reg cells must be set to 0. - - compatible: - enum: - - arm,arm710t - - arm,arm720t - - arm,arm740t - - arm,arm7ej-s - - arm,arm7tdmi - - arm,arm7tdmi-s - - arm,arm9es - - arm,arm9ej-s - - arm,arm920t - - arm,arm922t - - arm,arm925 - - arm,arm926e-s - - arm,arm926ej-s - - arm,arm940t - - arm,arm946e-s - - arm,arm966e-s - - arm,arm968e-s - - arm,arm9tdmi - - arm,arm1020e - - arm,arm1020t - - arm,arm1022e - - arm,arm1026ej-s - - arm,arm1136j-s - - arm,arm1136jf-s - - arm,arm1156t2-s - - arm,arm1156t2f-s - - arm,arm1176jzf - - arm,arm1176jz-s - - arm,arm1176jzf-s - - arm,arm11mpcore - - arm,armv8 # Only for s/w models - - arm,cortex-a5 - - arm,cortex-a7 - - arm,cortex-a8 - - arm,cortex-a9 - - arm,cortex-a12 - - arm,cortex-a15 - - arm,cortex-a17 - - arm,cortex-a32 - - arm,cortex-a34 - - arm,cortex-a35 - - arm,cortex-a53 - - arm,cortex-a55 - - arm,cortex-a57 - - arm,cortex-a65 - - arm,cortex-a72 - - arm,cortex-a73 - - arm,cortex-a75 - - arm,cortex-a76 - - arm,cortex-a77 - - arm,cortex-m0 - - arm,cortex-m0+ - - arm,cortex-m1 - - arm,cortex-m3 - - arm,cortex-m4 - - arm,cortex-r4 - - arm,cortex-r5 - - arm,cortex-r7 - - arm,neoverse-e1 - - arm,neoverse-n1 - - brcm,brahma-b15 - - brcm,brahma-b53 - - brcm,vulcan - - cavium,thunder - - cavium,thunder2 - - faraday,fa526 - - intel,sa110 - - intel,sa1100 - - marvell,feroceon - - marvell,mohawk - - marvell,pj4a - - marvell,pj4b - - marvell,sheeva-v5 - - marvell,sheeva-v7 - - nvidia,tegra132-denver - - nvidia,tegra186-denver - - nvidia,tegra194-carmel - - qcom,krait - - qcom,kryo - - qcom,kryo260 - - qcom,kryo280 - - qcom,kryo385 - - qcom,kryo468 - - qcom,kryo485 - - qcom,scorpion - - enable-method: - $ref: '/schemas/types.yaml#/definitions/string' - oneOf: - # On ARM v8 64-bit this property is required - - enum: - - psci - - spin-table - # On ARM 32-bit systems this property is optional - - enum: - - actions,s500-smp - - allwinner,sun6i-a31 - - allwinner,sun8i-a23 - - allwinner,sun9i-a80-smp - - allwinner,sun8i-a83t-smp - - amlogic,meson8-smp - - amlogic,meson8b-smp - - arm,realview-smp - - aspeed,ast2600-smp - - brcm,bcm11351-cpu-method - - brcm,bcm23550 - - brcm,bcm2836-smp - - brcm,bcm63138 - - brcm,bcm-nsp-smp - - brcm,brahma-b15 - - marvell,armada-375-smp - - marvell,armada-380-smp - - marvell,armada-390-smp - - marvell,armada-xp-smp - - marvell,98dx3236-smp - - marvell,mmp3-smp - - mediatek,mt6589-smp - - mediatek,mt81xx-tz-smp - - qcom,gcc-msm8660 - - qcom,kpss-acc-v1 - - qcom,kpss-acc-v2 - - renesas,apmu - - renesas,r9a06g032-smp - - rockchip,rk3036-smp - - rockchip,rk3066-smp - - socionext,milbeaut-m10v-smp - - ste,dbx500-smp - - ti,am3352 - - ti,am4372 - - cpu-release-addr: - $ref: '/schemas/types.yaml#/definitions/uint64' - - description: - Required for systems that have an "enable-method" - property value of "spin-table". - On ARM v8 64-bit systems must be a two cell - property identifying a 64-bit zero-initialised - memory location. - - cpu-idle-states: - $ref: '/schemas/types.yaml#/definitions/phandle-array' - description: | - List of phandles to idle state nodes supported - by this cpu (see ./idle-states.yaml). - - capacity-dmips-mhz: - $ref: '/schemas/types.yaml#/definitions/uint32' - description: - u32 value representing CPU capacity (see ./cpu-capacity.txt) in - DMIPS/MHz, relative to highest capacity-dmips-mhz - in the system. - - dynamic-power-coefficient: - $ref: '/schemas/types.yaml#/definitions/uint32' - description: - A u32 value that represents the running time dynamic - power coefficient in units of uW/MHz/V^2. The - coefficient can either be calculated from power - measurements or derived by analysis. - - The dynamic power consumption of the CPU is - proportional to the square of the Voltage (V) and - the clock frequency (f). The coefficient is used to - calculate the dynamic power as below - - - Pdyn = dynamic-power-coefficient * V^2 * f - - where voltage is in V, frequency is in MHz. - - power-domains: - $ref: '/schemas/types.yaml#/definitions/phandle-array' - description: - List of phandles and PM domain specifiers, as defined by bindings of the - PM domain provider (see also ../power_domain.txt). - - power-domain-names: - $ref: '/schemas/types.yaml#/definitions/string-array' - description: - A list of power domain name strings sorted in the same order as the - power-domains property. + $nodename: + const: cpus + + arm,pbha-performance-only: + $ref: /schemas/types.yaml#/definitions/uint8-array + description: PBHA values that only affect performance + minItems: 1 + maxItems: 15 + items: + maximum: 15 + + arm,pbha-no-aliases: + $ref: /schemas/types.yaml#/definitions/uint8-array + description: PBHA values that must only be used if all mappings have the + same value. + minItems: 1 + maxItems: 15 + items: + maximum: 15 - For PSCI based platforms, the name corresponding to the index of the PSCI - PM domain provider, must be "psci". - - qcom,saw: - $ref: '/schemas/types.yaml#/definitions/phandle' - description: | - Specifies the SAW* node associated with this CPU. - - Required for systems that have an "enable-method" property - value of "qcom,kpss-acc-v1" or "qcom,kpss-acc-v2" - - * arm/msm/qcom,saw2.txt - - qcom,acc: - $ref: '/schemas/types.yaml#/definitions/phandle' - description: | - Specifies the ACC* node associated with this CPU. - - Required for systems that have an "enable-method" property - value of "qcom,kpss-acc-v1" or "qcom,kpss-acc-v2" - - * arm/msm/qcom,kpss-acc.txt - - rockchip,pmu: - $ref: '/schemas/types.yaml#/definitions/phandle' - description: | - Specifies the syscon node controlling the cpu core power domains. - - Optional for systems that have an "enable-method" - property value of "rockchip,rk3066-smp" - While optional, it is the preferred way to get access to - the cpu-core power-domains. - - secondary-boot-reg: - $ref: '/schemas/types.yaml#/definitions/uint32' - description: | - Required for systems that have an "enable-method" property value of - "brcm,bcm11351-cpu-method", "brcm,bcm23550" or "brcm,bcm-nsp-smp". - - This includes the following SoCs: | - BCM11130, BCM11140, BCM11351, BCM28145, BCM28155, BCM21664, BCM23550 - BCM58522, BCM58525, BCM58535, BCM58622, BCM58623, BCM58625, BCM88312 - - The secondary-boot-reg property is a u32 value that specifies the - physical address of the register used to request the ROM holding pen - code release a secondary CPU. The value written to the register is - formed by encoding the target CPU id into the low bits of the - physical start address it should jump to. - -if: - # If the enable-method property contains one of those values - properties: - enable-method: - contains: - enum: - - brcm,bcm11351-cpu-method - - brcm,bcm23550 - - brcm,bcm-nsp-smp - # and if enable-method is present - required: - - enable-method - -then: - required: - - secondary-boot-reg - -required: - - device_type - - reg - - compatible - -dependencies: - rockchip,pmu: [enable-method] additionalProperties: true examples: - - | + -| + /{ cpus { - #size-cells = <0>; - #address-cells = <1>; + arm,pbha-performance-only = /bits/ 8 <0x01 0x05 0x09>; + arm,pbha-no-aliases = /bits/ 8 <0x02 0x04 0x06 0x08>; cpu@0 { - device_type = "cpu"; - compatible = "arm,cortex-a15"; - reg = <0x0>; - }; - - cpu@1 { - device_type = "cpu"; - compatible = "arm,cortex-a15"; - reg = <0x1>; - }; - - cpu@100 { - device_type = "cpu"; - compatible = "arm,cortex-a7"; - reg = <0x100>; - }; - - cpu@101 { - device_type = "cpu"; - compatible = "arm,cortex-a7"; - reg = <0x101>; - }; - }; - - - | - // Example 2 (Cortex-A8 uniprocessor 32-bit system): - cpus { - #size-cells = <0>; - #address-cells = <1>; - - cpu@0 { - device_type = "cpu"; - compatible = "arm,cortex-a8"; - reg = <0x0>; - }; - }; - - - | - // Example 3 (ARM 926EJ-S uniprocessor 32-bit system): - cpus { - #size-cells = <0>; - #address-cells = <1>; - - cpu@0 { - device_type = "cpu"; - compatible = "arm,arm926ej-s"; - reg = <0x0>; - }; - }; - - - | - // Example 4 (ARM Cortex-A57 64-bit system): - cpus { - #size-cells = <0>; - #address-cells = <2>; - - cpu@0 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x0>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@1 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x1>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100 { device_type = "cpu"; compatible = "arm,cortex-a57"; - reg = <0x0 0x100>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; + ... }; - cpu@101 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x101>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@10000 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x10000>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@10001 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x10001>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@10100 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x10100>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@10101 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x0 0x10101>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100000000 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x0>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100000001 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x1>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100000100 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x100>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100000101 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x101>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100010000 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x10000>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100010001 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x10001>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100010100 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x10100>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; - - cpu@100010101 { - device_type = "cpu"; - compatible = "arm,cortex-a57"; - reg = <0x1 0x10101>; - enable-method = "spin-table"; - cpu-release-addr = <0 0x20000000>; - }; }; + }; ... diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 143c2209368d6a5375c5356f0216b4b614c6d1b3..af3d833544ba38997838f25381d96f2b0e91ba6b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1691,6 +1691,26 @@ config ARM64_CNP at runtime, and does not affect PEs that do not implement this feature. +config ARM64_PBHA + bool "Enable support for Page Based Hardware Attributes (PBHA)" + default n + select ARCH_USES_HIGH_VMA_FLAGS + help + Page Based Hardware Attributes (PBHA) allow the SoC hardware to + change behaviour depending on which mapping was used to access + a page of memory. e.g. access via one mapping may always result + in the data being cached, whereas using another mapping of the same + physical memory. + + The behaviour of each PBHA bit is not defined. Say no unless you + are very sure you want this + + For PBHA_BIT0: It has the following features: + a) kernel pte entry will be set PBHA 59 bit during pagetable startup + b) Introduce VM_PBHA_BIT0 for feature use. + c) User pte entry will be set PBHA 59 bit for vma with VM_PBHA_BIT0. + d) Introduce /proc//pbha_bit0 to update whole user task. + endmenu menu "ARMv8.3 architectural features" diff --git a/arch/arm64/configs/openeuler_defconfig b/arch/arm64/configs/openeuler_defconfig index decbb9bf87748e1083ad86b0a0a8493d05a8cf50..45202a6d34ada234012d0121425be30dce06babc 100644 --- a/arch/arm64/configs/openeuler_defconfig +++ b/arch/arm64/configs/openeuler_defconfig @@ -478,6 +478,7 @@ CONFIG_ARM64_VHE=y CONFIG_ARM64_PMEM=y CONFIG_ARM64_RAS_EXTN=y CONFIG_ARM64_CNP=y +CONFIG_ARM64_PBHA=y # end of ARMv8.2 architectural features # diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index c7fe08dce2052d0ab45cb831ae3c340bb5caa725..ae9e88c28d43e9148c2b62465505fb7e7f86b76c 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -76,6 +76,9 @@ #define ARM64_HAS_WFXT 68 #define ARM64_WORKAROUND_HISILICON_ERRATUM_162100125 69 #define ARM64_HAS_LDAPR 70 +#define ARM64_HAS_PBHA 71 +#define ARM64_HAS_PBHA_STAGE1 72 +#define ARM64_HAS_PBHA_STAGE2 73 #define ARM64_NCAPS 80 diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index cce2f74ae618475c70a064b57636c27f462de42d..711f64fa1aa04428a571f58773fe17603d400f6b 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -78,6 +78,7 @@ struct arm64_ftr_reg { }; extern struct arm64_ftr_reg arm64_ftr_reg_ctrel0; +extern unsigned long arm64_pbha_stage2_safe_bits; int arm64_cpu_ftr_regs_traverse(int (*op)(u32, u64, void *), void *argp); @@ -767,6 +768,20 @@ static inline bool system_supports_tlb_range(void) cpus_have_const_cap(ARM64_HAS_TLB_RANGE); } +#ifdef CONFIG_ARM64_PBHA +extern bool pbha_enabled; + +static inline bool system_supports_pbha(void) +{ + return pbha_enabled; +} +#else +static inline bool system_supports_pbha(void) +{ + return false; +} +#endif + extern int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index cf35d1968c93df2e711803e79be40d108fe4f78b..05b56370ded5ad153afce2457c2719f696327783 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -126,6 +126,7 @@ #define VTCR_EL2_VS_SHIFT 19 #define VTCR_EL2_VS_8BIT (0 << VTCR_EL2_VS_SHIFT) #define VTCR_EL2_VS_16BIT (1 << VTCR_EL2_VS_SHIFT) +#define VTCR_EL2_PBHA_MASK GENMASK(28, 25) #define VTCR_EL2_T0SZ(x) TCR_T0SZ(x) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 8886d43cfb110713ea49f831a52bcd169782dbd1..f5dff6d40529d9a7957fada4c189242f1c739a4c 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -35,6 +35,10 @@ struct kvm_pgtable { * @KVM_PGTABLE_PROT_W: Write permission. * @KVM_PGTABLE_PROT_R: Read permission. * @KVM_PGTABLE_PROT_DEVICE: Device attributes. + * @KVM_PGTABLE_PROT_PBHA0: Page-Based Hardware Attribute 0. + * @KVM_PGTABLE_PROT_PBHA1: Page-Based Hardware Attribute 1. + * @KVM_PGTABLE_PROT_PBHA2: Page-Based Hardware Attribute 2. + * @KVM_PGTABLE_PROT_PBHA3: Page-Based Hardware Attribute 3. */ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_X = BIT(0), @@ -42,6 +46,11 @@ enum kvm_pgtable_prot { KVM_PGTABLE_PROT_R = BIT(2), KVM_PGTABLE_PROT_DEVICE = BIT(3), + + KVM_PGTABLE_PROT_PBHA0 = BIT(59), + KVM_PGTABLE_PROT_PBHA1 = BIT(60), + KVM_PGTABLE_PROT_PBHA2 = BIT(61), + KVM_PGTABLE_PROT_PBHA3 = BIT(62), }; #define PAGE_HYP (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W) diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index e3e28f7daf62bd69763e16ff45f9bcd4a07ca0fe..3c96193953e931492b7a23ff39a9afee39feb89e 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -5,6 +5,7 @@ #include #include #include +#include static inline unsigned long arch_calc_vm_prot_bits(unsigned long prot, unsigned long pkey __always_unused) @@ -17,6 +18,9 @@ static inline unsigned long arch_calc_vm_prot_bits(unsigned long prot, if (system_supports_mte() && (prot & PROT_MTE)) ret |= VM_MTE; + if (system_support_pbha_bit0() && (prot & PROT_PBHA_BIT0)) + ret |= VM_PBHA_BIT0; + return ret; } #define arch_calc_vm_prot_bits(prot, pkey) arch_calc_vm_prot_bits(prot, pkey) @@ -55,6 +59,9 @@ static inline pgprot_t arch_vm_get_page_prot(unsigned long vm_flags) if (vm_flags & VM_MTE) prot |= PTE_ATTRINDX(MT_NORMAL_TAGGED); + if (vm_flags & VM_PBHA_BIT0) + prot |= PROT_PBHA_BIT0; /* select PBHA BIT 0 for pbha */ + return __pgprot(prot); } #define arch_vm_get_page_prot(vm_flags) arch_vm_get_page_prot(vm_flags) @@ -70,6 +77,9 @@ static inline bool arch_validate_prot(unsigned long prot, if (system_supports_mte()) supported |= PROT_MTE; + if (system_support_pbha_bit0()) + supported |= PROT_PBHA_BIT0; + return (prot & ~supported) == 0; } #define arch_validate_prot(prot, addr) arch_validate_prot(prot, addr) diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 01a96d07ae741c58f90b3fc21f04776a449b058d..a5cff5b376f68b50abd9413d9253f6e3b1654914 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -146,6 +146,8 @@ #define PTE_CONT (_AT(pteval_t, 1) << 52) /* Contiguous range */ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ +#define PTE_PBHA_MASK (_AT(pteval_t, 0xf) << 59) /* Page Base Hardware Attributes */ +#define PTE_PBHA0 (_AT(pteval_t, 1) << 59) /* PBHA 59 bit */ #define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 @@ -260,6 +262,10 @@ #define TCR_TBI1 (UL(1) << 38) #define TCR_HA (UL(1) << 39) #define TCR_HD (UL(1) << 40) +#define TCR_HPD0 (UL(1) << 41) +#define TCR_HPD1 (UL(1) << 42) +#define TCR_HWU0nn_MASK (UL(0xf) << 43) +#define TCR_HWU1nn_MASK (UL(0xf) << 47) #define TCR_NFD0 (UL(1) << 53) #define TCR_NFD1 (UL(1) << 54) #define TCR_E0PD0 (UL(1) << 55) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index f2843785f2ec7021bb7bd2e25b7baaffb7e38e64..1999bda3be616d1994928592c3b5d93baa2b65b8 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -515,6 +515,32 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) __pgprot_modify(prot, PTE_ATTRINDX_MASK, \ PTE_ATTRINDX(MT_NORMAL_NC) | PTE_PXN | PTE_UXN) + +extern unsigned long arm64_pbha_perf_only_values; +static inline unsigned long __pbha_check_perf_only(unsigned long pbha_val) +{ + if (test_bit(pbha_val, &arm64_pbha_perf_only_values)) + return FIELD_PREP(PTE_PBHA_MASK, pbha_val); + return 0; +} + +#define pgprot_pbha(prot, pbha_val) \ + __pgprot_modify(prot, PTE_PBHA_MASK, __pbha_check_perf_only(pbha_val)) + +static inline pte_t pte_mkpbha(pte_t pte, unsigned long pbha_val) +{ + return set_pte_bit(pte, __pgprot(__pbha_check_perf_only(pbha_val))); +} + +#define pmd_mkpbha(pmd, pbha_val) pte_pmd(pte_mkpbha(pmd_pte(pmd), pbha_val)) + +static inline pte_t pte_rmpbha(pte_t pte, unsigned long pbha_val) +{ + return clear_pte_bit(pte, __pgprot(__pbha_check_perf_only(pbha_val))); +} + +#define pmd_rmpbha(pmd, pbha_val) pte_pmd(pte_rmpbha(pmd_pte(pmd), pbha_val)) + #define __HAVE_PHYS_MEM_ACCESS_PROT struct file; extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index 29bcb5bb45a3543bacf7568273a3ae5f0b54b5e7..fc56198ec28e52c452efa421101d29b643afbaf6 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -21,4 +21,7 @@ static inline bool arch_parse_debug_rodata(char *arg) } #define arch_parse_debug_rodata arch_parse_debug_rodata +void *get_early_fdt_ptr(void); +void early_fdt_map(u64 dt_phys); + #endif diff --git a/arch/arm64/include/uapi/asm/mman.h b/arch/arm64/include/uapi/asm/mman.h index 1e6482a838e11a29617672d93f5f4e72ea8f859f..af6ffde748f26c449d1c1491768d900285a560e0 100644 --- a/arch/arm64/include/uapi/asm/mman.h +++ b/arch/arm64/include/uapi/asm/mman.h @@ -6,5 +6,6 @@ #define PROT_BTI 0x10 /* BTI guarded page */ #define PROT_MTE 0x20 /* Normal Tagged mapping */ +#define PROT_PBHA_BIT0 0x40 /* PBHA 59 bit */ #endif /* ! _UAPI__ASM_MMAN_H */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 90700ce19e663395ee2cad42a1554f333f882f21..97744503704d077a8f965f6d300340cb1fd312b7 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -70,11 +70,16 @@ #include #include #include +#include #include +#include +#include +#include #include #include #include +#include #include #include #include @@ -84,6 +89,7 @@ #include #include #include +#include /* Kernel representation of AT_HWCAP and AT_HWCAP2 */ static unsigned long elf_hwcap __read_mostly; @@ -110,6 +116,10 @@ EXPORT_SYMBOL(arm64_use_ng_mappings); DEFINE_PER_CPU_READ_MOSTLY(const char *, this_cpu_vector) = vectors; +unsigned long __ro_after_init arm64_pbha_perf_only_values; +EXPORT_SYMBOL(arm64_pbha_perf_only_values); +unsigned long __ro_after_init arm64_pbha_stage2_safe_bits; + /* * Flag to indicate if we have computed the system wide * capabilities based on the boot time active CPUs. This @@ -1574,6 +1584,226 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, #endif + +#ifdef CONFIG_ARM64_PBHA +static u8 pbha_stage1_enable_bits; +static DEFINE_SPINLOCK(pbha_dt_lock); + +bool pbha_enabled; + +/* For the value 5, return a bitmap with bits 5, 4, and 1 set. */ +static unsigned long decompose_pbha_values(u8 val) +{ + int i; + unsigned long mask = 0; + + for (i = 1; i <= 15; i++) { + if ((i & val) == i) + set_bit(i, &mask); + } + + return mask; +} + +/* + * The bits of a value are safe if all values that can be built from those + * enabled bits are listed as only affecting performance. + * e.g. 5 would also need 1 and 4 to be listed. + * + * When there is a conflict with the bits already enabled, the new value is + * skipped. + * e.g. if 5 already caused bit-0 and bit-2 to be enabled, adding 3 to the list + * would need to test 7 as bit-2 is already enabled. If 7 is not listed, 3 is + * skipped and bit-1 is not enabled. + */ +static void stage2_test_pbha_value(u8 val) +{ + unsigned long mask; + + mask = decompose_pbha_values(val | arm64_pbha_stage2_safe_bits); + if ((arm64_pbha_perf_only_values & mask) == mask) + arm64_pbha_stage2_safe_bits |= val; +} + +void update_pbha_perf_only_bit(const u8 *bits, int cnt) +{ + u8 val; + int i; + + /* any listed value is usable at stage 1 */ + for (i = 0 ; i < cnt; i++) { + val = bits[i]; + if (val > 0xf) + continue; + + pbha_stage1_enable_bits |= val; + set_bit(val, &arm64_pbha_perf_only_values); + } +} + +static bool plat_can_use_pbha_stage1(const struct arm64_cpu_capabilities *cap, + int scope) +{ + u8 val; + struct device_node *cpus; + const u8 *perf_only_vals; + int num_perf_only_vals, i; + + if (!has_cpuid_feature(cap, scope)) + return false; + + /* + * Calls with scope == SCOPE_LOCAL_CPU need only testing whether this + * cpu has the feature. A later 'system' scope call will check for a + * firmware description. + */ + if (scope == SCOPE_LOCAL_CPU) + return true; + + spin_lock(&pbha_dt_lock); + if (pbha_enabled) + goto out_unlock; + + cpus = of_find_node_by_path("/cpus"); + if (!cpus) + goto done; + + perf_only_vals = of_get_property(cpus, "arm,pbha-performance-only", + &num_perf_only_vals); + if (!perf_only_vals) + goto done; + + update_pbha_perf_only_bit(perf_only_vals, num_perf_only_vals); + + /* + * for stage2 the values are collapsed back to 4 bits that can only + * enable values in the arm64_pbha_perf_only_values mask. + */ + for (i = 0 ; i < num_perf_only_vals; i++) { + val = perf_only_vals[i]; + if (val > 0xf) + continue; + + stage2_test_pbha_value(val); + } + +done: + of_node_put(cpus); + pbha_enabled = true; + +out_unlock: + spin_unlock(&pbha_dt_lock); + return !!pbha_stage1_enable_bits; +} + +static void enable_pbha_inner(void) +{ + u64 tcr; + + if (!pbha_stage1_enable_bits) + return; + + tcr = read_sysreg(tcr_el1); + tcr |= FIELD_PREP(TCR_HWU0nn_MASK, pbha_stage1_enable_bits); + tcr |= FIELD_PREP(TCR_HWU1nn_MASK, pbha_stage1_enable_bits); + tcr |= FIELD_PREP(TCR_HPD0, 1) | FIELD_PREP(TCR_HPD1, 1); + + write_sysreg(tcr, tcr_el1); + isb(); + local_flush_tlb_all(); +} + +static void cpu_enable_pbha(struct arm64_cpu_capabilities const *cap) +{ + enable_pbha_inner(); +} + +static inline bool cpu_has_pbha(void) +{ + u64 mmfr1 = read_cpuid(ID_AA64MMFR1_EL1); + int val = cpuid_feature_extract_unsigned_field(mmfr1, + ID_AA64MMFR1_HPD_SHIFT); + + return val == 2; +} + +void __init early_pbha_init(void) +{ + void *fdt; + int node; + const u8 *prop; + int size; + + spin_lock(&pbha_dt_lock); + + fdt = get_early_fdt_ptr(); + if (!fdt) + goto unlock; + + node = fdt_path_offset(fdt, "/cpus"); + if (node < 0) + goto unlock; + + prop = fdt_getprop(fdt, node, "arm,pbha-performance-only", &size); + if (!prop) + goto unlock; + + if (!cpu_has_pbha()) + goto unlock; + + update_pbha_perf_only_bit(prop, size); + enable_pbha_inner(); + + pbha_enabled = true; + + early_pbha_bit0_init(); + +unlock: + spin_unlock(&pbha_dt_lock); +} + +/* + * PBHA's behaviour is implementation defined, as is the way it combines + * stage1 and stage2 attributes. If the kernel has KVM supported, and booted + * at EL2, only these CPUs can allow PBHA in a guest, as KVM knows how the PBHA + * bits are combined. This prevents the host being affected by some + * implementation defined behaviour from the guest. + * + * The TRM for these CPUs describe stage2 as overriding stage1. + */ +static const struct midr_range pbha_stage2_wins[] = { + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A77), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78), + {}, +}; + +static bool plat_can_use_pbha_stage2(const struct arm64_cpu_capabilities *cap, + int scope) +{ + /* Booted at EL2? */ + if (!is_hyp_mode_available() && !is_kernel_in_hyp_mode()) + return false; + + if (!is_midr_in_range_list(read_cpuid_id(), cap->midr_range_list)) + return false; + + /* + * Calls with scope == SCOPE_LOCAL_CPU need only testing whether this + * cpu has the feature. A later 'system' scope call will check for a + * firmware description. + */ + if (scope == SCOPE_LOCAL_CPU) + return true; + + if (!__system_matches_cap(ARM64_HAS_PBHA_STAGE1)) + return false; + + return !!arm64_pbha_stage2_safe_bits; +} +#endif /* CONFIG_ARM64_PBHA */ + #ifdef CONFIG_ARM64_AMU_EXTN /* @@ -2295,6 +2525,34 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, .min_field_value = 1, }, + { + .capability = ARM64_HAS_PBHA, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .sys_reg = SYS_ID_AA64MMFR1_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR1_HPD_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = 2, + }, +#ifdef CONFIG_ARM64_PBHA + { + .desc = "Page Based Hardware Attributes (PBHA)", + .capability = ARM64_HAS_PBHA_STAGE1, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .sys_reg = SYS_ID_AA64MMFR1_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR1_HPD_SHIFT, + .matches = plat_can_use_pbha_stage1, + .min_field_value = 2, + .cpu_enable = cpu_enable_pbha, + }, + { + .capability = ARM64_HAS_PBHA_STAGE2, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = plat_can_use_pbha_stage2, + .midr_range_list = pbha_stage2_wins, + }, +#endif {}, }; diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 6e3f04b12bcb881d3cc01be7a1e3e295f0f1e7c4..5bc343fc2a91b0572acb4e6d32b654153729a3c0 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -451,11 +451,15 @@ SYM_FUNC_START_LOCAL(__primary_switched) #ifdef CONFIG_KASAN bl kasan_early_init +#endif + mov x0, x21 // pass FDT address in x0 + bl early_fdt_map // Try mapping the FDT early +#ifdef CONFIG_ARM64_PBHA + bl early_pbha_init // Init PBHA early via FDT #endif #ifdef CONFIG_RANDOMIZE_BASE tst x23, ~(MIN_KIMG_ALIGN - 1) // already running randomized? b.ne 0f - mov x0, x21 // pass FDT address in x0 bl kaslr_early_init // parse FDT for KASLR options cbz x0, 0f // KASLR disabled? just proceed orr x23, x23, x0 // record KASLR offset diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8d3fede89ae19848c8c3f5a2c67a1c3be0b40e2e..3a68772a63fb6f5ec066c874398478e0fb962c92 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -95,6 +95,9 @@ KVM_NVHE_ALIAS(gic_nonsecure_priorities); KVM_NVHE_ALIAS(__start___kvm_ex_table); KVM_NVHE_ALIAS(__stop___kvm_ex_table); +/* PBHA bits for stage2 */ +KVM_NVHE_ALIAS(arm64_pbha_stage2_safe_bits); + #endif /* CONFIG_KVM */ #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 9181d2856be339463e26d6abf10a7b36e61cbc23..d612ac8ae85565f4dda81dc98a1196ec4e7a0c73 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -19,6 +19,7 @@ #include #include #include +#include enum kaslr_status { KASLR_ENABLED, @@ -79,13 +80,12 @@ static __init const u8 *kaslr_get_cmdline(void *fdt) * containing function pointers) to be reinitialized, and zero-initialized * .bss variables will be reset to 0. */ -u64 __init kaslr_early_init(u64 dt_phys) +u64 __init kaslr_early_init(void) { void *fdt; u64 seed, offset, mask, module_range; const u8 *cmdline, *str; unsigned long raw; - int size; /* * Set a reasonable default for module_alloc_base in case @@ -99,8 +99,7 @@ u64 __init kaslr_early_init(u64 dt_phys) * and proceed with KASLR disabled. We will make another * attempt at mapping the FDT in setup_machine() */ - early_fixmap_init(); - fdt = fixmap_remap_fdt(dt_phys, &size, PAGE_KERNEL); + fdt = get_early_fdt_ptr(); if (!fdt) { kaslr_status = KASLR_DISABLED_FDT_REMAP; return 0; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 0d6cd99f73f57c067031862219500ec3161b0115..c687866612d9cee16a8fc21a48ced048a0dff553 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -185,6 +185,21 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } +static void *early_fdt_ptr __initdata; + +void __init *get_early_fdt_ptr(void) +{ + return early_fdt_ptr; +} + +asmlinkage void __init early_fdt_map(u64 dt_phys) +{ + int fdt_size; + + early_fixmap_init(); + early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); +} + static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size; diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 6f85c1821c3fbef3afdae0707ce700ba6a19115a..225cf4ffa108858e34d7563d37cda882d316b160 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -431,7 +431,7 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) { u64 vtcr = VTCR_EL2_FLAGS, mmfr0; u32 parange, phys_shift; - u8 lvls; + u8 lvls, pbha = 0xf; if (type & ~KVM_VM_TYPE_ARM_IPA_SIZE_MASK) return -EINVAL; @@ -474,6 +474,19 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type) */ vtcr |= VTCR_EL2_HA; + /* + * Enable PBHA for stage2 on systems that support it. The configured + * value will always be 0, which is defined as the safe default + * setting. On Cortex cores, enabling PBHA for stage2 effectively + * disables it for stage1. + * When the HAS_PBHA_STAGE2 feature is supported, clear the 'safe' + * bits to allow the guest's stage1 to use these bits. + */ + if (cpus_have_final_cap(ARM64_HAS_PBHA_STAGE2)) + pbha = pbha ^ arm64_pbha_stage2_safe_bits; + if (cpus_have_final_cap(ARM64_HAS_PBHA)) + vtcr |= FIELD_PREP(VTCR_EL2_PBHA_MASK, pbha); + /* Set the vmid bits */ vtcr |= (kvm_get_vmid_bits() == 16) ? VTCR_EL2_VS_16BIT : diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 99cd6e7184083d5a1f92d3151054e0c5e5b270ee..4effa2dd05185ed2c6698a6a0762d98271a2fe7d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -224,6 +224,8 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, */ WARN_ON(!pte_present(pte)); + pte = maybe_mk_pbha_bit0(pte, find_vma(mm, addr)); + if (!pte_cont(pte)) { set_pte_at(mm, addr, ptep, pte); return; diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 78b9e489d8f6943e1bb7a517e0ac81993956d01e..44c595550ae61a5fca60a58cfa1278414a73c1d2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include @@ -126,6 +127,10 @@ static bool pgattr_change_is_safe(u64 old, u64 new) */ pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; +#ifdef CONFIG_ARM64_PBHA + mask |= PTE_PBHA0; +#endif + /* creating or taking down mappings is always safe */ if (old == 0 || new == 0) return true; @@ -372,6 +377,8 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) return; + prot = pgprot_pbha_bit0(prot); + phys &= PAGE_MASK; addr = virt & PAGE_MASK; end = PAGE_ALIGN(virt + size); @@ -1152,6 +1159,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, { unsigned long addr = start; unsigned long next; + pgprot_t prot; pgd_t *pgdp; p4d_t *p4dp; pud_t *pudp; @@ -1180,7 +1188,10 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, if (!p) return -ENOMEM; - pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL)); + prot = __pgprot(PROT_SECT_NORMAL); + prot = pgprot_pbha_bit0(prot); + + pmd_set_huge(pmdp, __pa(p), prot); } else vmemmap_verify((pte_t *)pmdp, node, addr, next); } while (addr = next, addr != end); @@ -1300,6 +1311,7 @@ void __set_fixmap(enum fixed_addresses idx, ptep = fixmap_pte(addr); if (pgprot_val(flags)) { + flags = pgprot_pbha_bit0(flags); set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { pte_clear(&init_mm, addr, ptep); diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 807dc634bbd2464a993f33524cb495f9df2ba9f2..000819979b96f96477a1517433d2d4911b9b5528 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -151,6 +151,11 @@ static const struct prot_bits pte_bits[] = { .val = PTE_GP, .set = "GP", .clear = " ", + }, { + .mask = PTE_PBHA0, + .val = PTE_PBHA0, + .set = "PBHA0", + .clear = " ", }, { .mask = PTE_ATTRINDX_MASK, .val = PTE_ATTRINDX(MT_DEVICE_nGnRnE), diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c index 537db49c31b9230f2b64bb2cfaf60196ded631c1..8670654bf5618de42910e3d3d4fd5ef611dd64d5 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -23,6 +23,7 @@ bool efi_nokaslr = !IS_ENABLED(CONFIG_RANDOMIZE_BASE); bool efi_noinitrd; int efi_loglevel = CONSOLE_LOGLEVEL_DEFAULT; bool efi_novamap = IS_ENABLED(CONFIG_LOONGARCH); /* LoongArch call svam() in kernel */; +bool efi_pbha; static bool efi_nosoftreserve; static bool efi_disable_pci_dma = IS_ENABLED(CONFIG_EFI_DISABLE_PCI_DMA); @@ -234,6 +235,8 @@ efi_status_t efi_parse_options(char const *cmdline) efi_parse_option_graphics(val + strlen("efifb:")); } else if (!strcmp(param, "memmap") && val) { efi_parse_option_memmap(val); + } else if (!strcmp(param, "pbha")) { + efi_pbha = true; } } efi_bs_call(free_pool, buf); diff --git a/drivers/firmware/efi/libstub/fdt.c b/drivers/firmware/efi/libstub/fdt.c index d48b0de05b62fdab81b9b0a7033766a1ceac0e75..cb8e8e4e3f63bd492031f17e4ceb470cc5d86a5d 100644 --- a/drivers/firmware/efi/libstub/fdt.c +++ b/drivers/firmware/efi/libstub/fdt.c @@ -9,6 +9,7 @@ #include #include +#include #include #include "efistub.h" @@ -27,6 +28,59 @@ static void fdt_update_cell_size(void *fdt) fdt_setprop_u32(fdt, offset, "#size-cells", EFI_DT_SIZE_CELLS_DEFAULT); } +#ifdef CONFIG_ARM64_PBHA +extern bool efi_pbha; + +static efi_status_t fdt_init_hbm_mode(void *fdt, int node) +{ + efi_guid_t oem_config_guid = EFI_OEMCONFIG_VARIABLE_GUID; + unsigned long size; + efi_status_t efi_status; + u8 hbm_mode; + int status; + u8 fdt_val32; + u8 arr[16] = { 0x1 }; + + if (!efi_pbha) + goto out; + + efi_status = get_efi_var(L"HBMMode", &oem_config_guid, NULL, &size, + &hbm_mode); + if (efi_status != EFI_SUCCESS) + goto out; + + if (hbm_mode != HBM_MODE_CACHE) + goto out; + + fdt_val32 = 1; + status = fdt_setprop_var(fdt, node, "linux,pbha-bit0", fdt_val32); + if (status) + return EFI_LOAD_ERROR; + + node = fdt_subnode_offset(fdt, 0, "cpus"); + if (node < 0) { + node = fdt_add_subnode(fdt, 0, "cpus"); + if (node < 0) + return EFI_LOAD_ERROR; + } + + /* Current PBHA bit59 is need to enable PBHA bit0 mode. */ + status = fdt_setprop_var(fdt, node, "arm,pbha-performance-only", arr); + if (status) { + efi_err("PBHA: arm,pbha-performance-only failed\n"); + return EFI_LOAD_ERROR; + } + +out: + return EFI_SUCCESS; +} +#else +static inline efi_status_t fdt_init_hbm_mode(void *fdt, int node) +{ + return EFI_SUCCESS; +} +#endif + static efi_status_t update_fdt(void *orig_fdt, unsigned long orig_fdt_size, void *fdt, int new_fdt_size, char *cmdline_ptr, u64 initrd_addr, u64 initrd_size) @@ -148,6 +202,9 @@ static efi_status_t update_fdt(void *orig_fdt, unsigned long orig_fdt_size, } } + if (fdt_init_hbm_mode(fdt, node) != EFI_SUCCESS) + goto fdt_set_fail; + /* Shrink the FDT back to its minimum size: */ fdt_pack(fdt); diff --git a/drivers/soc/hisilicon/Makefile b/drivers/soc/hisilicon/Makefile index cbc01ad5b8fd3cd9557e4e67cf10c846e5106c8d..e0f966d1ed6dd6123645804f001f461e01e32b2b 100644 --- a/drivers/soc/hisilicon/Makefile +++ b/drivers/soc/hisilicon/Makefile @@ -3,3 +3,4 @@ obj-$(CONFIG_KUNPENG_HCCS) += kunpeng_hccs.o obj-$(CONFIG_HISI_HBMDEV) += hisi_hbmdev.o obj-$(CONFIG_HISI_HBMCACHE) += hisi_hbmcache.o +obj-$(CONFIG_ARM64_PBHA) += pbha.o diff --git a/drivers/soc/hisilicon/pbha.c b/drivers/soc/hisilicon/pbha.c new file mode 100644 index 0000000000000000000000000000000000000000..30d5057d8f9534f01d990309b32cc60b5250f3f0 --- /dev/null +++ b/drivers/soc/hisilicon/pbha.c @@ -0,0 +1,207 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) Huawei Technologies Co., Ltd. 2023. All rights reserved. + */ + +#define pr_fmt(fmt) "pbha: " fmt + +#include +#include +#include +#include +#include +#include +#include + +#include + +#define HBM_MODE_CACHE 1 + +bool __ro_after_init pbha_bit0_enabled; +bool __ro_after_init pbha_bit0_kernel_enabled; +static bool pbha_enabled_phase_1; + +void __init early_pbha_bit0_init(void) +{ + const u8 *prop; + void *fdt; + int node; + + /* Check whether PBHA is enabled or not. */ + if (!system_supports_pbha()) + return; + + fdt = get_early_fdt_ptr(); + if (!fdt) + return; + + node = fdt_path_offset(fdt, "/chosen"); + if (node < 0) + return; + + prop = fdt_getprop(fdt, node, "linux,pbha-bit0", NULL); + if (!prop) + return; + if (*prop == HBM_MODE_CACHE) + pbha_enabled_phase_1 = true; +} + +#define pte_pbha_bit0(pte) \ + (!!(pte_val(pte) & (PBHA_VAL_BIT0 << PBHA_BITS_SHIFT))) + +enum { + CLEAR_PBHA_BIT0_FLAG, + SET_PBHA_BIT0_FLAG, +}; + +static inline void pbha_bit0_update_pte_bits(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte, bool set) +{ + pte_t ptent = *pte; + + if (pte_present(ptent)) { + pte_t old_pte; + + old_pte = ptep_modify_prot_start(vma, addr, pte); + if (set) + ptent = pte_mkpbha(old_pte, PBHA_VAL_BIT0); + else + ptent = pte_rmpbha(old_pte, PBHA_VAL_BIT0); + ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); + } +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline void pbha_bit0_update_pmd_bits(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp, bool set) +{ + pmd_t pmd = *pmdp; + + if (pmd_present(pmd)) { + if (set) + pmd = pmd_mkpbha(pmd, PBHA_VAL_BIT0); + else + pmd = pmd_rmpbha(pmd, PBHA_VAL_BIT0); + + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } +} +#else +static inline void pbha_bit0_update_pmd_bits(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp, + bool set) +{ +} +#endif + +static int pbha_bit0_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) +{ + int *op = (int *)walk->private; + struct vm_area_struct *vma = walk->vma; + pte_t *pte, ptent; + spinlock_t *ptl; + bool set = (*op == SET_PBHA_BIT0_FLAG); + + ptl = pmd_trans_huge_lock(pmd, vma); + if (ptl) { + pbha_bit0_update_pmd_bits(vma, addr, pmd, set); + + spin_unlock(ptl); + return 0; + } + + if (pmd_trans_unstable(pmd)) + return 0; + + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + for (; addr != end; pte++, addr += PAGE_SIZE) { + ptent = *pte; + + pbha_bit0_update_pte_bits(vma, addr, pte, set); + } + pte_unmap_unlock(pte - 1, ptl); + cond_resched(); + return 0; +} + +static int pbha_bit0_test_walk(unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + struct vm_area_struct *vma = walk->vma; + + if (vma->vm_flags & VM_PFNMAP) + return 1; + + return 0; +} + +struct mm_walk_ops pbha_bit0_walk_ops = { + .pmd_entry = pbha_bit0_pte_range, + .test_walk = pbha_bit0_test_walk, +}; + +int pbha_bit0_update_vma(struct mm_struct *mm, int val) +{ + struct mmu_notifier_range range; + struct vm_area_struct *vma; + int old_val; + + if (!system_support_pbha_bit0()) + return -EINVAL; + + old_val = (mm->def_flags & VM_PBHA_BIT0) ? 1 : 0; + if (val == old_val) + return 0; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (val == SET_PBHA_BIT0_FLAG) { + mm->def_flags |= VM_PBHA_BIT0; + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (vma->vm_flags & VM_PBHA_BIT0) + continue; + vma->vm_flags |= VM_PBHA_BIT0; + vma_set_page_prot(vma); + } + } else { + mm->def_flags &= ~VM_PBHA_BIT0; + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (!(vma->vm_flags & VM_PBHA_BIT0)) + continue; + vma->vm_flags &= ~VM_PBHA_BIT0; + vma_set_page_prot(vma); + } + } + + inc_tlb_flush_pending(mm); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, 0, -1UL); + mmu_notifier_invalidate_range_start(&range); + walk_page_range(mm, 0, mm->highest_vm_end, &pbha_bit0_walk_ops, + &val); + mmu_notifier_invalidate_range_end(&range); + flush_tlb_mm(mm); + dec_tlb_flush_pending(mm); + + mmap_write_unlock(mm); + return 0; +} + +static int __init setup_pbha(char *str) +{ + if (!pbha_enabled_phase_1) + return 0; + + if (strcmp(str, "enable") == 0) { + pbha_bit0_enabled = true; + pbha_bit0_kernel_enabled = true; + } else if (strcmp(str, "user") == 0) { + pbha_bit0_enabled = true; + } + + pr_info("pbha bit_0 enabled, kernel: %d\n", pbha_bit0_kernel_enabled); + + return 0; +} +early_param("pbha", setup_pbha); diff --git a/fs/proc/base.c b/fs/proc/base.c index 24c70ff923b8ac046cec8b6f5d6de7ce939e7f11..2a4cc5c796c74ca5cd826fb1f668d39c4fd02f56 100644 --- a/fs/proc/base.c +++ b/fs/proc/base.c @@ -98,6 +98,7 @@ #include #include #include +#include #include #include "internal.h" #include "fd.h" @@ -1351,6 +1352,102 @@ static const struct file_operations proc_reliable_operations = { }; #endif +#ifdef CONFIG_ARM64_PBHA +static inline int pbha_bit0_check(struct task_struct *task, struct pid *pid) +{ + if (!system_support_pbha_bit0()) + return -EACCES; + + if (is_global_init(task)) + return -EACCES; + + if (!task->mm || (task->flags & PF_KTHREAD) || + (task->flags & PF_EXITING)) + return -EACCES; + + return 0; +} + +static ssize_t pbha_bit0_read(struct file *file, char __user *buf, + size_t count, loff_t *ppos) +{ + struct task_struct *task = get_proc_task(file_inode(file)); + struct pid *pid = proc_pid(file_inode(file)); + char buffer[PROC_NUMBUF]; + size_t len; + short val; + int err; + + if (!task) + return -ESRCH; + + err = pbha_bit0_check(task, pid); + if (err) { + put_task_struct(task); + return err; + } + + val = task->mm->def_flags & VM_PBHA_BIT0 ? 1 : 0; + put_task_struct(task); + len = snprintf(buffer, sizeof(buffer), "%hd\n", val); + return simple_read_from_buffer(buf, count, ppos, buffer, len); +} + +static ssize_t pbha_bit0_write(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + struct task_struct *task = get_proc_task(file_inode(file)); + struct pid *pid = proc_pid(file_inode(file)); + char buffer[PROC_NUMBUF]; + struct mm_struct *mm; + int val, err; + + if (!task) + return -ESRCH; + + err = pbha_bit0_check(task, pid); + if (err) + goto out; + + memset(buffer, 0, sizeof(buffer)); + if (count > sizeof(buffer) - 1) + count = sizeof(buffer) - 1; + if (copy_from_user(buffer, buf, count)) { + err = -EFAULT; + goto out; + } + + err = kstrtoint(strstrip(buffer), 0, &val); + if (err) + goto out; + if (val != 0 && val != 1) { + err = -EINVAL; + goto out; + } + + mm = get_task_mm(task); + if (!mm) { + err = -ENOENT; + goto out; + } + + err = pbha_bit0_update_vma(mm, val); + if (err) + count = -EINTR; + + mmput(mm); +out: + put_task_struct(task); + return err < 0 ? err : count; +} + +static const struct file_operations proc_pbha_bit0_ops = { + .read = pbha_bit0_read, + .write = pbha_bit0_write, + .llseek = generic_file_llseek, +}; +#endif + #ifdef CONFIG_AUDIT #define TMPBUFLEN 11 static ssize_t proc_loginuid_read(struct file * file, char __user * buf, @@ -3483,6 +3580,9 @@ static const struct pid_entry tgid_base_stuff[] = { #ifdef CONFIG_MEMORY_RELIABLE REG("reliable", S_IRUGO|S_IWUSR, proc_reliable_operations), #endif +#ifdef CONFIG_ARM64_PBHA + REG("pbha_bit0", 0644, proc_pbha_bit0_ops), +#endif #ifdef CONFIG_AUDIT REG("loginuid", S_IWUSR|S_IRUGO, proc_loginuid_operations), REG("sessionid", S_IRUGO, proc_sessionid_operations), @@ -3902,6 +4002,9 @@ static const struct pid_entry tid_base_stuff[] = { #ifdef CONFIG_MEMORY_RELIABLE REG("reliable", S_IRUGO|S_IWUSR, proc_reliable_operations), #endif +#ifdef CONFIG_ARM64_PBHA + REG("pbha_bit0", 0644, proc_pbha_bit0_ops), +#endif #ifdef CONFIG_AUDIT REG("loginuid", S_IWUSR|S_IRUGO, proc_loginuid_operations), REG("sessionid", S_IRUGO, proc_sessionid_operations), diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 9182d0c6d22c6b579e57f562cf93f4aa6231082d..d54e0e3474cc5ade169071c498bba58e3f9a7609 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -670,6 +670,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_MTE)] = "mt", [ilog2(VM_MTE_ALLOWED)] = "", #endif +#ifdef CONFIG_ARM64_PBHA + [ilog2(VM_PBHA_BIT0)] = "p0", +#endif #ifdef CONFIG_ARCH_HAS_PKEYS /* These come out via ProtectionKey: */ [ilog2(VM_PKEY_BIT0)] = "", diff --git a/include/linux/mm.h b/include/linux/mm.h index 34400f90933515196db1ac912345ca2a4168c2c1..a5316ffd4e1fb91450d2f533f76dfc0ce7be0016 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -392,6 +392,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_SHARE_POOL VM_NONE #endif +#if defined(CONFIG_ARM64_PBHA) +# define VM_PBHA_BIT0 VM_HIGH_ARCH_2 /* Page Base Hardware Attributes 4 bit*/ +#else +# define VM_PBHA_BIT0 VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif @@ -451,7 +457,7 @@ static inline bool arch_is_platform_page(u64 paddr) #define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB) /* This mask defines which mm->def_flags a process can inherit its parent */ -#define VM_INIT_DEF_MASK VM_NOHUGEPAGE +#define VM_INIT_DEF_MASK (VM_NOHUGEPAGE | VM_PBHA_BIT0) /* This mask is used to clear all the VMA flags used by mlock */ #define VM_LOCKED_CLEAR_MASK (~(VM_LOCKED | VM_LOCKONFAULT)) diff --git a/include/linux/pbha.h b/include/linux/pbha.h new file mode 100644 index 0000000000000000000000000000000000000000..25cb88aa3d1c3a33c8c36ec675359ed8b4649856 --- /dev/null +++ b/include/linux/pbha.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) Huawei Technologies Co., Ltd. 2023. All rights reserved. + */ +#ifndef __LINUX_PBHA_H +#define __LINUX_PBHA_H + +#include +#include +#include + +#define PBHA_VAL_BIT0 1UL +#define PBHA_BITS_SHIFT 59 + +#define EFI_OEMCONFIG_VARIABLE_GUID \ + EFI_GUID(0x21f3b3c5, 0x946d, 0x41c1, 0x83, 0x8c, 0x19, 0x4e, 0x48, \ + 0xaa, 0x41, 0xe2) + +#define HBM_MODE_MEMORY 0 +#define HBM_MODE_CACHE 1 + +#ifdef CONFIG_ARM64_PBHA +extern bool __ro_after_init pbha_bit0_enabled; +extern bool __ro_after_init pbha_bit0_kernel_enabled; +extern struct mm_walk_ops pbha_bit0_walk_ops; +extern void __init early_pbha_bit0_init(void); +extern int pbha_bit0_update_vma(struct mm_struct *mm, int val); + +static inline bool system_support_pbha_bit0(void) +{ + return pbha_bit0_enabled; +} + +static inline pgprot_t pgprot_pbha_bit0(pgprot_t prot) +{ + if (!system_support_pbha_bit0() || !pbha_bit0_kernel_enabled) + return prot; + + return pgprot_pbha(prot, PBHA_VAL_BIT0); +} + +static inline pte_t maybe_mk_pbha_bit0(pte_t pte, struct vm_area_struct *vma) +{ + if (!system_support_pbha_bit0()) + return pte; + + /* global init task will update pbha bit0 iff kernel can do this */ + if (unlikely(is_global_init(current)) && pbha_bit0_kernel_enabled && + !(vma->vm_flags & VM_PBHA_BIT0)) + vma->vm_flags |= VM_PBHA_BIT0; + + if (vma->vm_flags & VM_PBHA_BIT0) + pte = pte_mkpbha(pte, PBHA_VAL_BIT0); + + return pte; +} +#else +static inline bool system_support_pbha_bit0(void) { return false; } +static inline pgprot_t pgprot_pbha_bit0(pgprot_t prot) { return prot; } +static inline pte_t maybe_mk_pbha_bit0(pte_t pte, struct vm_area_struct *vma) +{ + return pte; +} +#endif + +#endif diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h index 66c408ccc6c662a0bacba4ad8474df19707036f4..4d23e72a26032f8195d7be0dbf7419554d19552b 100644 --- a/include/uapi/asm-generic/mman-common.h +++ b/include/uapi/asm-generic/mman-common.h @@ -13,6 +13,7 @@ #define PROT_SEM 0x8 /* page may be used for atomic ops */ /* 0x10 reserved for arch-specific use */ /* 0x20 reserved for arch-specific use */ +/* 0x40 reserved for arch-specific use */ #define PROT_NONE 0x0 /* page can not be accessed */ #define PROT_GROWSDOWN 0x01000000 /* mprotect flag: extend change to start of growsdown vma */ #define PROT_GROWSUP 0x02000000 /* mprotect flag: extend change to end of growsup vma */ diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index c9e363906530d030f26b6786570abe0c98c2ae7f..00ffc2dbbf2f7a71f78e4895fcfb433835a63d42 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -260,4 +260,6 @@ struct prctl_mm_map { #define PR_SET_MEMORY_MERGE 67 #define PR_GET_MEMORY_MERGE 68 + +#define PR_UPDATE_PBHA_BIT0 0x82312f0 #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/sys.c b/kernel/sys.c index 15efe4667397fc90760e6640b9cdac8deb361a80..68885eabad0618aa2ea956d024813b862cd24118 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2550,6 +2550,16 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, error = !!test_bit(MMF_VM_MERGE_ANY, &me->mm->flags); break; +#endif +#ifdef CONFIG_ARM64_PBHA + case PR_UPDATE_PBHA_BIT0: + if (arg3 || arg4 || arg5) + return -EINVAL; + if (arg2 != 0 && arg2 != 1) + return -EINVAL; + + error = pbha_bit0_update_vma(me->mm, arg2); + break; #endif default: error = -EINVAL; diff --git a/mm/memory.c b/mm/memory.c index 5893c178251a03881d2d14eb67dda50ca822f7f4..55d4375d4b279f9361b40f57225bfc82cde9e4a9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -74,6 +74,7 @@ #include #include #include +#include #include @@ -2969,6 +2970,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) entry = mk_pte(new_page, vma->vm_page_prot); entry = pte_sw_mkyoung(entry); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = maybe_mk_pbha_bit0(entry, vma); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -3709,6 +3711,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) __SetPageUptodate(page); entry = mk_pte(page, vma->vm_page_prot); + entry = maybe_mk_pbha_bit0(entry, vma); entry = pte_sw_mkyoung(entry); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); @@ -4016,6 +4019,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct page *page) inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page)); page_add_file_rmap(page, false); } + entry = maybe_mk_pbha_bit0(entry, vma); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); /* no need to invalidate: a not-present page won't be cached */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e27cd716ca95c9c46b9734e4170da63d3cf7a89e..caba5659d137f7a8f480db128260a331d0efc206 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include #include @@ -307,6 +308,8 @@ int vmap_range(unsigned long addr, unsigned long end, { int err; + prot = pgprot_pbha_bit0(prot); + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); flush_cache_vmap(addr, end); @@ -549,6 +552,8 @@ static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, WARN_ON(page_shift < PAGE_SHIFT); + prot = pgprot_pbha_bit0(prot); + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || page_shift == PAGE_SHIFT) return vmap_small_pages_range_noflush(addr, end, prot, pages);