You are super welcome, lovely.
It brings Ada and I a whole heap of pleasure running these instances and it's largely knowing that we're making a difference to our users, and providing a safe space for you all to grow and flourish that makes it all worth it for us.
I have been watching my love tie herself in knots over the last several days, having to deal with the drama that has been brought on, trying her best to bring everyone back together.
There's been bad behaviour from both sides, and I'm really disappointed to see that some of the worst of it came from our users, who didn't keep to the moral high ground, disregarded our instance rules and stoopped to levels of behaviour worse than that leveled against them.
There have been accusations against us (or Ada specifically) that we are a safe harbour for bad behaviour and cause harm to trans people through our inaction.
This is perhaps the cruelest accusation they could have leveled at Ada, as she works tirelessly to maintain a safe space for our community, and while I was hoping, for all the effort that she was investing into this issue, that she could make it work despite my own reservations, this last attack on her impeccable morality has made me very angry.
I'm sorry for those that wanted to remain federated, sorry that it came to this, but I am glad it's over now, purely for the mental health of my precious beloved.
Okay, so that was way more painful than expected... /sigh
So, one thing I'd mention is the systems and admin work involved in running an instance.
This is on top of the community moderation, and involves networking with other instance admins, maintaining good relations, deciding who to defeferate from, dealing with unhappy users, etc.
Then there's the setup and maintenance of the servers, security, hacks, DDoSing, backups, redundancy, monitoring, downtime, diagnosis, fixing performance issues, patching, coding, upgrades etc.
I wouldn't be here doing this without @ada. We make a formidable team, and without any self effacement, we are both at the top of our respective roles with decades of experience.
Big communities also magnify the amount of work involved. We're almost at the point where we are starting to consider getting additional people involved.
Moreover we're both here for the long haul, with the willingness and ability to personally cover the shortfall in hosting costs.
I'm not trying to convince you to stay here. But in addition to free hardware, you're going to need a small staff to do these things for you, so my advice is to work out if you have reliable AND trustworthy people (because these people will have access to confidential user data) who are committed to do this work long term with you. Where will you be in 3 years, 5, 10?
To be clear, $3k is an accurate, but unacceptable amount.
As in that's what it's actually costing us, but it's not what it should be costing. I'd imagine more like $250 is what we should be paying if I wasn't using AWS in the silly way I am.
I'm admitting up front that I've been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.
Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.
And so I didn't notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that's what got me.
About half that $3k is just data transfer costs.
Anyhow the notice was just to let our users know what is going on and that there'll be some maintenance windows in their future so it doesn't surprise anyone.
We have a plan and it will all work out.
Don't panic or have any kneejerk reactions, it's just an FYI.
Just want to say, I don't blame anyone else but myself.
I certainly don't blame anyone at 196.
I hope I'm really clear about that. It's one of the reasons I specifically didn't name 196 in my announcement.
We've got a solution planned, we've already started to implement it and have the image transfer issue solved already.
We can afford to cover this ridiculous AWS bill, I just need to do some maintenance work so this doesn't continue because I can't continue to line Jeff Bezos' pockets like this indefinitely.
Migration complete.
It wasn't an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately publicUrl is not a valid URL, so lemmy's /api/v3/site metadata endpoint started returning an error relative URL without a base instead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.
I have a process which copies emojos from the hajkey blahaj.zone site to our lemmy.
Last night (for me) it copied an invalid emojo into the database and it broke lemmy for hours while Ada and I were sleeping.
I've found the issue and fixed it now, so everything should be purring along once again, but for those who were left without the instance to use, I sincerely apologise.
How do you know I haven't always been the hacker who's in control? :D
Hi everyone.
Lemmy had an XSS vulnerability and we were one of the instances attacked through this vulnerability. We had our homepage replaced by a youtube video.
The lemmy devs were quick to create a PR for this issue and we have patched our instance to fix it,
Also while I do not believe any login tokens were compromised, we have taken the additional measure of rotating our secret key, so everyone will have to login to the site again as your existing credentials will no longer be valid.
If login fails to work properly, you may have to clear your cookies for the site... it seems to get confused about whether you're logged in or not.
Sorry that this happened, if you have any questions please contact @ada@lemmy.blahaj.zone or myself.
Database upsize completed. Still working on 0.18.1 upgrade, will be one further (software) restart in about an hour and then we'll be done and running the latest release candidate.
Okay, I'm taking down the server now to upsize thew instance hardware.
Migration has been completed!