Skip to content

gsTexManager: stop evicting when a texture cannot fit in VRAM - #109

Open
kaisa847 wants to merge 1 commit into
ps2dev:masterfrom
kaisa847:fix/texmanager-too-large
Open

kaisa847 wants to merge 1 commit into
ps2dev:masterfrom
kaisa847:fix/texmanager-too-large

Conversation

@kaisa847

Copy link
Copy Markdown

Problem

gsKit_TexManager_bind() → _blockAlloc() frees textures with a growing weight limit until a free block is large enough. The loop has no exit condition: when the texture is larger than all the VRAM the manager owns (4 MiB minus the frame and z buffers), or when gsKit_TexManager_init() was never called (__head is NULL), the loop keeps incrementing the weight forever and the program hangs.

Every texture goes through gsKit_TexManager_bind() — for example SDL binds every texture that way — so a single oversized texture freezes the whole application.

Fix

Count the textures still resident in VRAM on each pass. When a pass finds nothing left to free and no block fits, _blockAlloc() returns NULL. gsKit_TexManager_bind() then handles NULL: it prints an error, clears tex->Vram/tex->VramClut and returns without uploading, instead of hanging.

Testing

Tested on real hardware: a PSX DESR-5100 (PS2-based), with a test program started via ps2link. A 256×256 CT32 texture (256 KiB) is bound through a GSGLOBAL whose free VRAM is only 64 KiB. gsKit_TexManager_bind() runs in a separate low-priority thread; the main thread waits up to 2 s.

  • Unpatched gsKit: the bind thread never returns (the main thread reports haengt after 2 s and has to terminate it).
  • This PR: the bind returns, prints gsKit: TexManager: texture does not fit in VRAM, and leaves tex->Vram = 0.

🤖 Generated with Claude Code

_blockAlloc() frees textures with a growing weight limit until a free
block is large enough. The loop has no exit: when the texture is larger
than all VRAM the manager owns (4 MiB minus frame and z buffers), or
when gsKit_TexManager_init() was never called (__head is NULL), the
loop keeps incrementing the weight forever.

The program hangs in gsKit_TexManager_bind(), e.g. SDL binds every
texture through it, so one oversized SDL texture freezes the app.

Count the textures that are still in VRAM during each pass. When a pass
finds nothing left to free and no block fits, return NULL. Make
gsKit_TexManager_bind() handle NULL: print an error, clear tex->Vram and
tex->VramClut and return 0 without uploading.

Tested on real hardware (PSX DESR-5100 via ps2link): binding a
256x256 texture into 64 KiB of free VRAM hangs unpatched and returns
with an error message with this change.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant