Being less strict with memory allocations

[JJS]

Some games request a tad more memory for their heap than can be satisfied. Examples for that are Kurok (and all other games on the same engine) and OpenTTD (as recently noted here).

They always show the same symptoms, the debug log ends with multiple failed calls to sceKernelAllocPartitionMemory like this line repeatedly:

Code: Select all

call to sceKernelAllocPartitionMemory partitionId: 2, name: block, type:0, size:19456000, addr:0x00000000
failed with result: 0x800200D9

A simple way to get these games to start is to lower the amount of memory until the allocation succeeds. Assuming that a game doesn't really need every last byte of the requested memory this should work without negatively affecting performance. Of course this is transparent to the homebrew, so in case all memory is actually needed it is possible that the homebrew overwrites memory outside the allocation.


For my test I used R105 with a modified _hook_sceKernelAllocPartitionMemory function:

Code: Select all

SceUID _hook_sceKernelAllocPartitionMemory(SceUID partitionid, const char *name, int type, SceSize size, void *addr)
{
    tGlobals * g = get_globals();
    LOGSTR5("call to sceKernelAllocPartitionMemory partitionId: %d, name: %s, type:%d, size:%d, addr:0x%08lX\n", partitionid, (u32)name, type, size, (u32)addr);
    sceKernelWaitSema(g->memSema, 1, 0);

	// Try to allocate the requested memory. If the allocation fails due to an insufficient
	// amount of free memory try again with 10kB less until the allocation succeeds.
	// Don't allow to go under 80 % of the initial amount of memory.
	SceUID uid;
	SceSize original_size = size;
	do 
	{
		uid = sceKernelAllocPartitionMemory(partitionid,name, type, size, addr);
		if (uid <= 0)
		{
			// Allocation failes, check if we can go lower for another try
			if ((size > 10000) && ((size - 10000) > original_size - (original_size / 5)))
			{
				// Try again with 1kB less
				size -= 10000;
			}
			else
			{
				// Limit reached, break out of loop
				break;
			}
		}
	}
    while (uid <= 0);

	LOGSTR3("-> final allocation made for %d of %d requested bytes with result 0x%08lX\n", size, original_size, uid);

	if (uid > 0)
	{
        /***********************************************************************/
        /* Succeeded OS alloc.  Record the block ID in the tracking list.      */
        /* (Don't worry if there's no space to record it, we'll just have to   */
        /* leak it).                                                           */
        /***********************************************************************/
        if (g->osAllocNum < MAX_OS_ALLOCS)
        {
            g->osAllocs[g->osAllocNum] = uid;
            g->osAllocNum ++;
            LOGSTR1("Num tracked OS blocks now: %08lX\n", g->osAllocNum);
        }
        else
        {
            LOGSTR0("!!! EXCEEDED OS ALLOC TRACKING ARRAY\n");
        }
    }
    sceKernelSignalSema(g->memSema, 1);
    return uid;
}

For OpenTTD on Patapon 2 with R105 nonids I get this:
requested: 19456000 byte, granted: 19252000 byte -> delta of ~200kB

For Kurok with the same setup the situation is this:
requested: 18874368 byte, granted: 18338368 byte -> delta of ~500kB


Thoughts on this? On the one hand it will allow some previously broken homebrews to start, on the other hand there is no guarantee that they will run without error.

Advertising

[m0skit0]

Nice idea, I like it. I prefer having more homebrews start and eventually crash later rather than not starting at all.

Also another idea: if there's not enough memory, could we activate p5 and allocate from there? The game won't notice the difference, but this would mean no power-off/suspend available until the p5 allocs are freed. This can be fixed probably by registering HBL callbacks for power-off/suspend and closing p5 accordingly.

Advertising

[JJS]

I see some problems with allocating memory on p5:
P5 is only 4 MiB in size and the usual homebrew will have exactly one allocation for the heap when starting. If the heap is only 4 MiB it will always fit in p2, if it is larger it would have to be split across the partitions. But a game will just use the libc/newlib malloc implementation on the allocated junk of memory and this expects a continous block of memory as does the hombrew. I don't see how we could redirect memory access in any practical (and not-perfomance-killing way). Also calling any utility dialog overwrites p5 memory, so even in that case the memory would have to be swapped out to the memory stick I guess.

[m0skit0]

I thought previously about a virtual memory system by software, but without the proper hardware support (which PSP lacks) it would be awfully slow and hard to program so I ruled it out early. We can discuss it if you wish.

[JJS]

I would be interested to hear how you would go about implementing this in software.

At the moment I would imagine such a system to work like an emulator. Because you don't have hardware to check memory addresses of the executed program I think it would be necessary to parse and interpret the program in software.

Alternatively it should also be possible to rewrite the program and insert hooking code every time there is a memory access. So if the program would like to perform an e.g. "load word" instruction you would rewrite it to call a function that checks the address, if needed pages memory in/out, then finally performs the requested instruction and returns. Should be doable I guess but the performance would be unacceptable as you say.

[m0skit0]

Yes, the idea I came with was the second method you're talking about, which would be definitely much faster than the first one you mention, but still too slow to be done by software only.

[wololo]

there used to be more than 23MB of free Ram when HBL was loaded, how comes we can only give 19MB? The remaining 4MB are taken by the binary of the homebrew? In that case, using p5 for the binary might be an acceptable solution in some cases.

We can also try to load HBL a bit higher in ram if possible? if we can get 100kB back maybe it's worth it... I'm surprised that kurok lacks 500kB, I would expect HBL to "eat" 200kB tops (binary + operation + variables), but it means that we have at least 500kB less than what a cfw can offer...

Also, we should consider loading homebrews at 0x08804000 instead of 0x08900000, that might help a lot. I actually tried that a while ago, and ran into loads of problems. But as far as I know, CFW loads homebrews at 0x08804000. the noobz eLoader also had a setting variable to be able to set this loading value on a per homebrew basis.

That being said, I like your current solution and I suggest you sumbit it to the SVN.

[JJS]

About loading the homebrew at a lower address:
Might help, but maybe not as much as one would think because utility modules are loaded in the space between sceKernelLibrary and 0x08900000 now. Those would then be loaded above the homebrew as is the case with official games (and I guess on CFW).

Where the RAM goes to:
Mostly network modules I think. Especially games that use both adhoc and infrastructure should be starved for memory since HBL loads both modules when loading the homebrew instead of only loading them when they are needed. Actually thinking about it, a delayed loading of the utility modules when they are requested by the homebrew should give some breathing room. Of course the utility loading might fail then if there is not enough memory.

HBL load address:
If we would move the homebrew loading further down, how about loading HBL directly below where the homebrew is being relocated to? That way it is safe from being overwritten by some stack. But this might interfere with non-PRX (static) homebrews I guess.


I will submit that code soon. Edit: Submitted in R107.