312

[Update: Failed again] Update to 0.18.1-rc.1 tried and rolled back (lemmy.world)

submitted 2 years ago* (last edited 2 years ago) by ruud@lemmy.world to c/lemmyworld@lemmy.world

194 comments fedilink hide all child comments

We've upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can't login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn't work.

I'll create an issue for the devs and retry once it's fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won't be on 0.18.1 any time soon I'm afraid..

top 50 comments

sorted by: hot top controversial new old

[-] MrPear@lemmy.world 59 points 2 years ago

Thank you for the transparancy!

[-] GatoB@lemmy.world 6 points 2 years ago

I love post like this, another reason of why this instance is so good

[-] morelikepinniped@lemmy.world 25 points 2 years ago

Thanks for all your hard work @ruud@lemmy.world. It's not easy!

[-] Jerti@lemmy.world 15 points 2 years ago

UI: 0.18.0 BE: 0.18.1-rc.1 👍

seems to be working for me, I was already logged on

[-] ulu_mulu@lemmy.world 12 points 2 years ago

is that users can’t login anymore

Ouch, that's bad :(

Thank you for your work, as always :)

load more comments (1 replies)

[-] Machefi@lemmy.world 12 points 2 years ago

Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?

[-] ruud@lemmy.world 13 points 2 years ago

Because of a few things, really anoying bugs and the Jerboa app not working properly with older versions

[-] twistedtxb@lemmy.world 7 points 2 years ago

I suspect there will be a large influx of new users in two days, and that having Jerboa not working on .world might cause a few issues

[-] antik@lemmy.world 6 points 2 years ago

For example the frontpage is no longer being constantly filled with random posts when you set the filter to "active" or "top day".

[-] axzxc1236@lemmy.world 4 points 2 years ago* (last edited 2 years ago)

Reasons I can think of:

The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.
Sorting is broken pre 0.18, new posts keeps flowing in.
Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)

load more comments (15 replies)

load more comments (4 replies)

[-] OsrsNeedsF2P@lemmy.ml 11 points 2 years ago

Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!

Dessalines:

Nutomic:

load more comments (3 replies)

[-] slimerancher@lemmy.world 8 points 2 years ago

Oh. Good luck with the next try!

[-] necropola@lemmy.world 7 points 2 years ago* (last edited 2 years ago)

Everything (UI) is extremely slow now. Not an improvement at all. Also I'm frequently getting error messages like this now:

load more comments (6 replies)

[-] snargledorf@lemmy.world 7 points 2 years ago

@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?

With all the performance issues with 0.18.1, it's highlighted that there needs to be a way to stress test these updates before applying them to the main instance.

load more comments (6 replies)

[-] Guy_Fieris_Hair@lemmy.world 6 points 2 years ago

Suddenly my Jerboa won't change from local and I can't change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.

[-] sockenklaus@sh.itjust.works 3 points 2 years ago

What you're describing is an issue introduced with Jerboa 0.0.36-alpha and has been fixed in today released 0.0.37-alpha.

load more comments (4 replies)

[-] NABDad@lemmy.world 5 points 2 years ago

Thanks for the info. You're doing a great job!

[-] rrobin@lemmy.world 5 points 2 years ago

As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.

We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.

load more comments (1 replies)

[-] Asudox@lemmy.world 4 points 2 years ago

I am glad y'all rolled back. I had just enabled 2FA and could not login to my account. Rolling back removed 2FA so I was able to log back in. Thanks.

[-] ruud@lemmy.world 2 points 2 years ago

Which 2FA app did you use? I heard Authy didn't work

load more comments (1 replies)

[-] Spitz@lemmy.world 4 points 2 years ago

Jerboa crashes constantly for me now, doesn't even let me view anything. JSYK!

load more comments (4 replies)

[-] ekZepp@lemmy.world 4 points 2 years ago

[-] naneek@lemmy.world 4 points 2 years ago* (last edited 2 years ago)

Thanks for the transparency. Maybe it’s a good idea to have a test instance and some test cases/validation done there before updating the main instance. This is a regular process in any software/tech company/stack.

Testing should never be done directly on the prod instance.

[-] 65gmexl3@lemmy.world 3 points 2 years ago

i think they tested it which worked fine, but it is totally different when there are users online

[-] antik@lemmy.world 4 points 2 years ago

Correct there was a test instance and that worked fine but once the live instance was upgraded everything became very slow. Can't really emulate that kind of load

[-] Nintendo@lemmy.world 4 points 2 years ago

appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?

[-] Anarch157a@lemmy.world 4 points 2 years ago

Seems 0.18 is up... and...

OH NO ! The interface changed ! Bring it back ! Bring it back... /s

Just kidding... It's looking good, so it's time to explore a little and find all the new stuff.

Nice work @Rudd, thanks for all the hard work.

[-] rr7@lemmy.world 4 points 2 years ago

My current session works now with 0.18.0 + 0.18.1.rc1

[-] DRx@lemmy.world 4 points 2 years ago

Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.

Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though

any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?

[-] BlueN1te@lemmy.world 3 points 2 years ago

pretty brutal bug ngl. was hoping I'd be using jerboa again soon

[-] i_simp_4_tedcruz@lemmy.world 2 points 2 years ago* (last edited 2 years ago)

Is the bug with Jerboa that no subscribed communities are showing up? From my point of view logged in now I only have access to lemmy.world communities

load more comments (1 replies)

[-] xtract@lemmy.world 3 points 2 years ago

Working fine from Lift-off!

[-] flubba86@lemmy.world 3 points 2 years ago* (last edited 2 years ago)

I had a weird issue after it upgraded. My browser (Firefox on Android) showed the Lemmy UI but showed no posts. I checked my profile to see if I was logged in, and it showed I was logged in as someone else! I cleared my cache and refreshed, it showed I wasn't logged in. After I logged in it was fine, and showed my expected posts and my actual profile. In Liftoff app my account worked fine the whole time, didn't need to log in again.

[-] ren@lemmy.world 3 points 2 years ago

Good luck! May the software gods bless you with a functioning website with minimal bugs!

load more comments (1 replies)

[-] Sterben@lemmy.world 3 points 2 years ago

I am grad I didn't write my last post during the upgrade process.

Maybe next time, give a warning, or maintenance notice.

Thanks for trying though.

load more comments (1 replies)

[-] necrxfagivs@lemmy.world 3 points 2 years ago* (last edited 2 years ago)

Great! I'm getting a bug, when I upvote a comment or post, the "loading animation" takes forever to complete. If I refresh the upvote is there. It reminds me to the bug we had with comments and new posts.

Edit: Is the same for comments.

[-] Smokeless7048@lemmy.world 3 points 2 years ago

Thank you for the hard work Ruud!

[-] FlavorGrain@lemmy.world 2 points 2 years ago

Sucks that lemmy.world won't be working with Jerboa when the reddit apps quit working in a couple days.

[-] 65gmexl3@lemmy.world 3 points 2 years ago

alternatives like LiftOff is available

https://play.google.com/store/apps/details?id=com.liftoffapp.liftoff&pli=1

load more comments (8 replies)

[-] Shadywack@lemmy.world 2 points 2 years ago

Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!

load more comments