In Progress CPANEL-42516 - User doesn't exist in cache keys - don't understand error

[jimlongo]

Hi, does anyone have any clue what this error is referring to?

It seems to be generated the first time I invoke execute($user) in the controller called from the index.live.php page
I know I have the $user ('thursday1') in this case.
If I refresh the page it works as you would normally expect.

PHP Warning:  file_get_contents(/var/cpanel/caches/_generated_header_files/cache_keys.json): failed to open stream: Permission denied in /usr/local/cpanel/php/WHM.php on line 61
PHP Fatal error:  Uncaught Exception: User does not exist in cache_keys. Try regenerating the cache. in /usr/local/cpanel/php/WHM.php:66
Stack trace:
#0 /usr/local/cpanel/php/WHM.php(47): WHM::getUserCacheKey('thursday3')
#1 /usr/local/cpanel/php/WHM.php(10): WHM::processHeader('TCS...', 0, 0, NULL)
#2 /usr/local/cpanel/whostmgr/docroot/cgi/Tcs/View/Layout.php(6): WHM::header('TCS...', 0, 0)
#3 /usr/local/cpanel/whostmgr/docroot/cgi/Tcs/View/View.php(36): require_once('/usr/local/cpan...')
#4 /usr/local/cpanel/whostmgr/docroot/cgi/Tcs/Model/Tool/InstantFailMessage.php(15): Tcs\View\View->render('Blocks/abort')
#5 /usr/local/cpanel/whostmgr/docroot/cgi/Tcs/Model/Api/ResponseValidator.php(33): Tcs\Model\Tool\InstantFailMessage::create('Whm api: Authen...')
#6 /usr/local/cpanel/whostmgr/docroot/cgi/Tcs/Model/Api/AbstractApi.php(25): Tcs\Model\Api\ResponseValidator->validate(Object(stdClass), false)
#7 /usr/local/cpanel/base/frontend/jupiter/Tcs/Controller/YPageController.php(113): Tcs\Model\Api\Abstract in /usr/local/cpanel/php/WHM.php on line 66

don't know if it helps

cat /var/cpanel/caches/_generated_header_files/cache_keys.json
{"root":"1239935361"}

 

[cPRex]

Hey there! Can you let us know what specific page is causing that error to happen? It looks like an area of the WHM interface, but I'd like to be sure before I recommend anything.

 

[jimlongo]

Hi cPRex,

Thanks for responding, this is in the cPanel plugin page.
I'm on my `index.live.php` page.
I think it happens right after it echoes the header (well I see the header on the page, and the rest is blank)
it instantiates my controller, and then runs my override of execute()
I build the $header and $footer and some other variables, then like this ...

echo $header;
$controller = new MyPageController();
$controller->execute( $active_username , $allowed_free_domains );
echo $footer;

as it churns through my controller, I get to a place where I call your API to ForceSSLRedirect.
using this endpoint

/json-api/cpanel?cpanel_jsonapi_apiversion=3&cpanel_jsonapi_module=SSL&cpanel_jsonapi_func=toggle_ssl_redirect_for_domains&cpanel_jsonapi_user='. $this->user .'&domains='. $this->domain . '&state=1';

the response to that is

Whm api: AuthenticationValidator2
Whm api: AuthenticationValidator3 Array
   (
       [0] => The event UAPI::SSL::toggle_ssl_redirect_for_domains was handled successfully.
   )

then the page dies, and the error happens.

 

[cPRex]

Would you be able to check and see if the pages that are created on the fly are 600 instead of 644 after just that first load?

 

[jimlongo]

I don't understand, "pages that are created"
The files all exist on the server, are you saying there are more created?

 

[cPanelThomas]

The script which populates the cache_keys.json file is `scripts/rebuild_whm_chrome`. Re-run that and see if that fixes the problem. That error message could definitely be better, but I'm pretty sure that's what it is referring to in this context.

 

[jimlongo]

Hi same result.
Does the permissions warning lead to the fatal error?

PHP Warning:  file_get_contents(/var/cpanel/caches/_generated_header_files/cache_keys.json): failed to open stream: Permission denied in /usr/local/cpanel/php/WHM.php on line 61
PHP Fatal error:  Uncaught Exception: User does not exist in cache_keys. Try regenerating the cache. in /usr/local/cpanel/php/WHM.php:66
Stack trace:

# ll /var/cpanel/caches/_generated_header_files/
total 1228
drwx------  2 root root   4096 Mar  6 16:17 .
drwx--x--x. 6 root root   4096 Mar  6 16:17 ..
-rw-------  1 root root 310969 Mar  6 16:17 3740202532_0_0.html
-rw-------  1 root root 310754 Mar  6 16:17 3740202532_0_1.html
-rw-------  1 root root 310969 Mar  6 16:17 3740202532_1_0.html
-rw-------  1 root root 310754 Mar  6 16:17 3740202532_1_1.html
-rw-------  1 root root     21 Mar  6 16:17 cache_keys.json

 

[cPanelThomas]

Sounds highly likely to me. Probably why Rex was concerned about permissions. I know we've had a bug that sounded simliar to that open for a while regarding non-root WHM users and template compiles, though in a slightly different scope.

May just need to tweak `/var/cpanel/caches/_generated_header_files/cache_keys.json` to be 0644 permissions instead of 0600 to fix the problem *for now*. If so it's probably the same kind of permissions issue as CPANEL-15585 (though in *that* case it regards files within /var/cpanel/template_compiles).

Certainly it appears to be 0600 on some of my personal testing servers, so that would certainly explain the problem to me.

 

[jimlongo]

Thank you.
Changed permissions, reran the script.
Oops, that changes the permissions back to 600.
Changed permissions again ...



Still the same result ... but there's more info in the error log

[2023-03-06 16:37:06 +0000] info [cpanel] ---debug_hooks---
[2023-03-06 16:37:06 +0000] info [cpanel]             msg: Beginning execution of script hook.
[2023-03-06 16:37:06 +0000] info [cpanel]         context: Cpanel::UAPI::SSL::toggle_ssl_redirect_for_domains
[2023-03-06 16:37:06 +0000] info [cpanel]           stage: post
[2023-03-06 16:37:06 +0000] info [cpanel]          result: 0
[2023-03-06 16:37:06 +0000] info [cpanel]           point: main
[2023-03-06 16:37:06 +0000] info [cpanel]              id: 38eec658-d9c7-466e-afce-9027cbb7daeb
[2023-03-06 16:37:06 +0000] info [cpanel]            hook: /usr/local/cpanel/3rdparty/wp-toolkit/bin/handle-event
[2023-03-06 16:37:06 +0000] info [cpanel]   escalateprivs: 1
[2023-03-06 16:37:06 +0000] info [cpanel]          weight: 100
[2023-03-06 16:37:06 +0000] info [cpanel]            data: {"user":"xyzsitebuilder","args":{"state":"1","cpanel_jsonapi_func":"toggle_ssl_redirect_for_domains","cpanel_jsonapi_module":"SSL","domains":["d3daf0dd.xyzsitebuilder.xyz"],"cpanel_jsonapi_apiversion":"3","cpanel_jsonapi_user":"xyzsitebuilder"},"result":[1]}
[2023-03-06 16:37:06 +0000] info [cpanel] STDOUT output from hook: /usr/local/cpanel/3rdparty/wp-toolkit/bin/handle-event
[2023-03-06 16:37:06 +0000] info [cpanel] 1 The event UAPI::SSL::toggle_ssl_redirect_for_domains was handled successfully.
[2023-03-06 16:37:06 +0000] info [cpanel] End STDOUT from hook
[2023-03-06 16:37:06 +0000] info [cpanel] ---debug_hooks---
[2023-03-06 16:37:06 +0000] info [cpanel]             msg: Finished execution of script hook.
[2023-03-06 16:37:06 +0000] info [cpanel]         context: Cpanel::UAPI::SSL::toggle_ssl_redirect_for_domains
[2023-03-06 16:37:06 +0000] info [cpanel]           stage: post
[2023-03-06 16:37:06 +0000] info [cpanel]          result: 1
[2023-03-06 16:37:06 +0000] info [cpanel]           point: main
[2023-03-06 16:37:06 +0000] info [cpanel]              id: 38eec658-d9c7-466e-afce-9027cbb7daeb
[2023-03-06 16:37:06 +0000] info [cpanel]            hook: /usr/local/cpanel/3rdparty/wp-toolkit/bin/handle-event
[2023-03-06 16:37:06 +0000] info [cpanel]   escalateprivs: 1
[2023-03-06 16:37:06 +0000] info [cpanel]          weight: 100
[2023-03-06 16:37:06 +0000] info [cpanel]            data: {"user":"xyzsitebuilder","args":{"state":"1","cpanel_jsonapi_func":"toggle_ssl_redirect_for_domains","cpanel_jsonapi_module":"SSL","domains":["d3daf0dd.xyzsitebuilder.xyz"],"cpanel_jsonapi_apiversion":"3","cpanel_jsonapi_user":"xyzsitebuilder"},"result":[1]}
PHP Warning:  file_get_contents(/var/cpanel/caches/_generated_header_files/cache_keys.json): failed to open stream: Permission denied in /usr/local/cpanel/php/WHM.php on line 61
PHP Fatal error:  Uncaught Exception: User does not exist in cache_keys. Try regenerating the cache. in /usr/local/cpanel/php/WHM.php:66

 

[cPanelThomas]

Looked into the code (/usr/local/cpanel/Whostmgr/Templates/Command.pm in case you are curious), and it is creating the directory for the file 0700 with root owner. Even correcting permissions on the individual file likely won't solve it because we don't have exec on the dir.

I think either we need to be setting the permissions on these files to something world readable OR using different cache dirs for the various WHM resellers and setting the owner if we need to keep these permissions strict for security reasons. Either way, this is probably a bug, so I've filed CPANEL-42516 on it.

Overall the design seems easy enough to correct -- we're storing the filename by "cache key" in that json file regarding what file to pick out of `/var/cpanel/caches/_generated*` directories. Considering we always will know REMOTE_USER in this context, we may as well just index it that way via subdirectories with the correct permissions. That should fix the problem (and be slightly faster, as we'd reduce the amount of files we load by 1 in that case due to already knowing the answer of "where is my compiled template").

 

[jimlongo]

Thanks for the response.
I'm pretty much broken until this is fixed.

I had seen this error before in testing, but it was always under some unusual testing circumstance.
Now it happens every time I invoke execute()

I think going from PHP 7 to 8 caused this for my plugin.
I went from PHP7 on a Centos7 box, to PHP8 on AlmaLinux.

Just as a test of your theory (I imagine there's more to it than this)
I changed the permissions of the directory and file and tried again.

Didn't get the PHP Warning: file_get_contents
But did get the Fatal Error following.

[[email protected] caches]# ll
drwx--x--x.   6 root root   4096 Mar  6 16:36 .
drwx--x--x. 114 root root  12288 Mar  6 23:05 ..
drwxr-xr-x.   2 root root  81920 Feb 22 19:48 _build_translated_js_files
lrwxrwxrwx.   1 root root     21 Feb 22 19:57 Cpanel-OS -> 'linux|almalinux|8|7|0'
drwx------    2 root root   4096 Mar  6 16:36 _generated_command_files
drwx------    2 root root   4096 Mar  6 16:36 _generated_footer_files
drwxr-xr-x    2 root root   4096 Mar  6 16:36 _generated_header_files
-rw-------    1 root root 177836 Feb 23 22:20 manage_plugins.11.108.0.11
-rw-------    1 root root 197412 Feb 27 23:13 manage_plugins.11.108.0.12
-rw-r--r--.   1 root root    546 Mar  6 03:01 store_licenses.json.cache

[[email protected] caches]# ll _generated_header_files/cache_keys.json
-rw-r--r-- 1 root root 21 Mar  6 16:36 _generated_header_files/cache_keys.json

 

[cPanelThomas]

As far as I'm aware, PHP7/8 differences shouldn't really matter here. The only thing that should matter is what user whostmgrd is executing as (who is REMOTE_USER). If it is root, everything should be fine. If it is a non-root reseller, everything should fail due to permissions being incorrect to load the file. It makes sense that you'd stop seeing the first warning after permissions were corrected in that light. As to why it doesn't have that reseller in the cache keys file, that's probably because an earlier paste showed that only root had a cache_key set. Not sure why that was, but *normally* that's what regenerate_whm_chrome should address. Fixing the permissions issue by obviating the need to lookup that file should additionally fix that issue (just use REMOTE_USER to look it up).

Possibly you could work around this by adding an entry to the cache_keys file for the user in question (just set the id to the same one root has) for now, but I have not tested this.

Anyways, I wish I had an ETA on fixes for you, but I just can't give one right now due to other cases I'm still working on. If someone else doesn't pick up the case I'll get to it after those.

 

[jimlongo]

Thanks, understood.

The only other piece of information i can add (I'm testing some of your hypotheses) is what happens after I refresh, and execute() can finish

I then inspected the cache_keys.json file and find that it has not changed.

$  cat _generated_header_files/cache_keys.json
{"root":"1688438057"}

 

[jimlongo]

one more ... tested this suggestion

Possibly you could work around this by adding an entry to the cache_keys file for the user in question (just set the id to the same one root has) for now, but I have not tested this.

# cat _generated_header_files/cache_keys.json
{"root":"1688438057","xyzsitebuilder":"1688438057"}

this has no effect, same error.

 

[jimlongo]

as it churns through my controller, I get to a place where I call your API to ForceSSLRedirect.

I think the error about cache keys may be a red herring caused by my workflow.
I think the reason it's getting thrown is because my ResponseValidators are tripping over this "errors" in an otherwise successful call.


call

/json-api/cpanel?cpanel_jsonapi_apiversion=3&cpanel_jsonapi_user=permission2&cpanel_jsonapi_module=SSL&cpanel_jsonapi_func=toggle_ssl_redirect_for_domains&domains=subdomain.xyz.xyz&state=1

$response

{
    "module": "SSL",
    "apiversion": 3,
    "result": {
        "messages": null,
        "status": 1,
        "metadata": {
            "transformed": 1
        },
        "errors": [
            "The event UAPI::SSL::toggle_ssl_redirect_for_domains was handled successfully."
        ],
        "data": [
            1
        ],
        "warnings": null
    },
    "func": "toggle_ssl_redirect_for_domains"
}

At least I can confirm that when I remove this call i DO NOT get the cache keys error and can proceed.

I will look at tweaking my Response Validator, but isn't it weird that the API would send an errors value when the call has a "status" : 1 ?