-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Random invalid session and inconsistent service accounts #19510
Comments
@pschichtel Do you only have one KES server for production? |
@jiuker production has 3, backup has 1 |
Does the 3 KES server have the same keys for production? @pschichtel |
They are all connected to the same vault (with a dedicated V2 KV engine for minio), so I'd assume so. How can I check? |
Could you check the key |
Not sure what you mean |
Check for overlapping value assignments between two clients |
Sorry for being confused!
By key, do you mean a KES key or an access key/secret access key? There is no "site-replicator-0" KES key, so I assume access key. What do you mean by "value" then?
What do you mean by "value assignments"? And what clients? I just checked with mcli again (
|
I only found a strange case where when I open two minio login pages in one browser at the same time, there have one said that |
@jiuker I have the occasional case, where the page after login stays blank, because I think you case sounds like a race condition on the shared cookies/localStorage/sessionStorage between the browser tabs, that are not shared between browsers. |
Yeah. Will return back to login page for |
@jiuker I don't think it is limited to a specific page, I've seen it happen on several different pages.
I'm not so sure anymore, because I get errors with mcli too, that doesn't go through the console, right? |
We can't reproduce any of the issues reported here. |
How can I properly clear replication settings from both sites? then I could test the production cluster without site replication and see if that helps. |
|
I just noticed, even the replication rules on buckets are completely inconsistent from refresh to refresh. @poornas thanks, I'll try that next week. |
Bucket versioning is also affected. it seems like everything somehow related to site replication is completely inconsistent between the nodes of the production cluster. It also seems to have gotten worse since I last checked last week. |
@poornas I removed the backup site from replication and it's all fine now. Should the |
Remove |
Are you saying it should automatically disappear after removing site-replication? Because it hasn't so far, neither in the production site nor in the backup site. Both sites don't have any other replication rules. So I'll delete the service accounts to have a clean state. |
Yeah. It should disappear. If not, you can try delete it for I didn't reproduce your case. |
I removed the accounts, I'll upgrade both instances to latest now and then setup replication again in the evening. |
I remember @harshavardhana saying something about this in a past issue: The /v1/service-accounts endpoint is rather slow (400-900 ms "wait" time in the browser) given that this is a small cluster (5 nodes) and only 3 service accounts exist and my connection is basically local, this feels noticeable slow in the UI. This is still the case even after disabling replication. Is the timing within a normal range or would this be worth investigating? I originally thought this is caused by the replication problem, but apparently it isn't. |
@harshavardhana now after the upgrade to I wonder if this is something caused by the upgrade process of the operator? Should I open a new issue for this? |
At least it seems to be limited to the console API, running |
@harshavardhana after upgrading to I'm now convinced that that the upgrade as performed by the operator seems to cause or at least worsen this issue. Should I create a new issue? possibly over at minio/console ? |
This generally I don't it to occur unless someone actively wipes your credentials. |
Can you collect both sites all their backend .minio.sys folder and share it with us ? |
@harshavardhana Seems there is quite a bit of information in there that I don't think I can just share. Is it possible to limit the requested files? Otherwise I'd first have to clear internally if it's ok to share this stuff. What I noticed while poking around:
|
Ha... I found the offender. I slowly, one-by-one went through the pods (from last to first similar to the sts controller), deleted them and let the sts controller recreate them. between each pod I repeatedly checked the the /service-accounts endpoint. pods 4, 3 and 2 did nothing, restarting the pod 1 completely resolved the issue. |
Do you have logs from this pod before deleting it ? |
I do, but I don't think there was anything of interest. I'll check... |
here you go: production-1-logs.txt I noticed that the cluster once lost the quorum. The log file btw includes both the update and my restart the fixed the problem. |
Minor update on this: Apparently this happened again at some point in the last 3 weeks. Sadly this time I don't know which node was affected, but the upgrade to yesterdays's release resolved the issue for now (probably because all nodes got restarted). I'll keep monitoring. |
I once again had this after the upgrade to the newest release. Restarting the nodes one by one once again solved it. |
I suspect this is due to #19905 and has been around since 03-28 release |
Timing-wise that sounds reasonable. Will the fix help immediately with the next upgrade or will it require me to cycle the deployment one more time? |
Just upgraded to |
@pschichtel Don't do rolling upgrades. |
@klauspost how am I supposed to do upgrades with the operator? |
Is there any documentation on the topic of upgrades in kubernetes with the operator at all? The operator seems to just do a StatefulSet update which in turn triggers a rolling upgrade last to first (that's k8s' default strategy iirc). The main documentation on upgrades doesn't seem appliable to a) containerized deployments and b) kubernetes deployments, given that overwriting a binary in a container isn't really useful, when the overwrite is lost after restarting the process. Also does this apply only to rolling upgrades or to rolling restarts in general ? because I'm pretty sure I triggered this issue with just a rolling restart on the exact same version. |
Operator upgrades first the binary inside the container via pushing the binary directly to the pods, once the pod is restarted() it proceeds to updating the container image which does rolling. If the first step fails, I suspect the operator is still doing a statefulset update. |
So your operator logs will have some information regarding this. |
I'll can check that |
I've seen this line both in this production multi-node setup as well as in a homelab deployment also using the operator:
|
yeah this means upgrade would have failed. |
this is the exact reason why you face this problem. |
I'd argue the operator would better stop and enter an error state, when a correct upgrade is not possible. How can we find the cause of the tenant:
image:
repository: quay.io/minio/minio
tag: 'RELEASE.2024-06-11T03-13-30Z'
name: tenant
configuration:
name: credentials
pools:
- name: main
servers: 5
volumesPerServer: 1
storageClassName: ''
size: 123123123
labels:
velero.io/exclude-from-backup: "true"
metrics:
enabled: true
certificate:
requestAutoCert: true
env:
- name: MINIO_OPERATOR_TLS_ENABLE
value: "off"
- name: MINIO_DOMAIN
value: "minio.example.org"
- name: MINIO_BROWSER_REDIRECT_URL
value: "https://console.minio.example.org"
- name: MINIO_SERVER_URL
value: "https://minio.example.org"
log:
disabled: true
prometheus:
disabled: true
prometheusOperator: true |
also there no NetworkPolicy and also no outbound firewall in general, so there is no reason why the operator shouldn't be able to download and distribute the minio binaries. |
This is a follow up to #19217 and #19201
After my vacation I just verified the state of the minio installation again after the previous issues.
Expected Behavior
Once logged in I'd expect not to randomly receive "invalid session" warnings or to get randomly logged out when navigating to certain pages (e.g. the Site Replication config page).
I would also expect to the same service accounts on my root user every time I refresh the Access Keys page (or when directly accessing
/api/v1/service-accounts
).Current Behavior
I randomly get invalid session responses ("The Access Key Id you provided does not exist in our records.") from the backend and on some pages, that leads to a redirect to the login page.
I also get a different list service accounts every time I refresh, sometimes it doesn't even include the site-replicator-0 account, which would explain why I'm still seeing #19217. Actually in my tests now by refreshing
/api/v1/service-accounts
a bunch of times, I rarely get all 4 service accounts.The backup site still occasionally logs this as in #19217:
Steps to Reproduce (for bugs)
I'm still not sure how I arrived at this state, I assume by enabling site replication.
I've checked that KES is working on both the production and the backup site. At this point I'm not even able to disable site replication on the production site, because I get constantly logged out (redirected to login page) from the page.
The single-node backup instance does not observe this behavior. there, I never get invalid session responses, I always get the same 4 service accounts on the root user (including site-replicator-0) and I can also access the Site Replication page.
Context
It makes using the minio console difficult. I assume, replication from backup to production would not reliably work (or be a lot slower), but that's not currently something I need to do.
Interestingly
mcli admin user svcacct list production admin
always returns the complete list of service accounts for my root user, although not always in the same order, but that doesn't matter. S3 clients in general don't seem to be affected, at least not functionally.To elaborate on the setup:
2 sites:
The keys between the KES deployments are identical (replicated files from production site can be decrypted on backup site. The production KES setup is responsive and can successfully access the vault (I created and deleted a test key to confirm).
Your Environment
minio --version
):RELEASE.2024-04-06T05-26-02Z
uname -a
):Linux 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: