I was browsing 196 recently and I noticed that stuff between slrpnk and blahaj weren't federating between the 2 properly.

From slrpnk (another lemmy instance) there looks like there hasn't been a post in 196 since yesterday but checking from blahaj it's as alive as always.

I then checked from lemmy.world, beehaw, sopuli.xyz, and lemmy.zip to see if it was just an issue with slrpnk but it appears to be widespread as all of those other instances have the same issue.

Hopefully me making a post here isn't an issue (and that it federates properly) but I figured reporting a bug wouldn't be an issue.

Sorry if this isn't allowed/in the wrong place, but I wanted it to have a little visibility in case others found themselves in a position to chip in. Recovering from a breach is painful and soaks up resources that could be used elsewhere.

I mostly lurk and read Blahaj from a different instance, but when I saw the hacked announcement it reminded me what a great resource and community is here and I figured now would be a good time as any to let the Admins know I appreciate their efforts!

Link to their kofi: https://ko-fi.com/blahajzone (but to be security conscious, you shouldn't click that, you should really go get the link from their sidebar :) ).

cross-posted from: https://lemmy.ca/post/66634564

Cloudflare: Why I left lemmy.world to join another instance

Sad to see. We should all know cloudlfare decrypts our data before it reaches the service it protects. A perfect surveillance man-in-the-middle.

I was happy to find a home in the fediverse that uses Anubis instead.

https://anubis.techaro.lol/

This version again includes a couple of security fixes. Thanks to the people who found and reported them!

The first one in particular requires manual action from instance admins. Lemmy’s default Nginx config uses $proxy_add_x_forwarded_for to set the X-Forwarded-For header, which does not override existing values. So clients can spoof the IP and bypass rate limits. The solution is to use $remote_addr instead. If you use Ansible this will be changed automatically during the upgrade, otherwise you will need to do it manually. If you dont use Nginx, ensure that any X-Forwarded-For headers sent by the client are overwritten.

The remaining security vulnerabilities are in the Lemmy code itself, and will be fixed simply by upgrading.

Security:

• Rate limit bypass via X-Forwarded-For header spoofing in actix-web ConnectionInfo
• Login Endpoint User Enumeration via HTTP Response Code and Timing
• Blocked users can edit private messages sent before the block
• Lower-ranked federated moderator can remove higher-ranked moderators via federation
• Featuring post over federation does not validate community nor write modlog
• Stored XSS via markdown image alt-text in lemmy-ui html5-embed##

Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.

The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn't use postgres). From what we can tell, the attacker did not do anything with that access, so we don't believe any user data was accessed, but we can't be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.

Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.

We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it's as smooth as we think it will be, but we can't be sure!

As stated earlier, our writefreely instance is still up and running as it wasn't impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.

We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don't plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won't be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn't seem worth the cost/effort to run it isolated.

in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked

Edit - Piefed is back now!

pressing "save" in my settings page does nothing, not even show a popup telling me something failed. it's been that way for a few months at least, since april of this year i think.

i've tried modifying and saving my profile in other clients, and it seems to work there! i can change my bio and username without issues.

however, something i can't do, in any client, is change my profile picture, which is unfortunate since mine has been broken* for some time. in the default lemmy ui, as i said, pressing the save button does nothing. however, i've also tried to change my profile picture in tesseract and blorp, and both complain about an invalid url when trying to save the profile pic. both can save other settings tho, and image upload works fine, as the images i try to set appear in my uploads tab. using the js console, i can see that changing the profile pic does a PUT https://lemmy.blahaj.zone/api/v3/user/save_user_settings request, which fails (error 400). trying to change the banner image has the same problem. this happens no matter the browser or device i try this on.

is this also a problem for other people?

* (i've since removed my profile pic in blorp, which is why i've got the default lemmy profile pic now instead of a broken image icon)

(i'm sorry if this kind of post is not allowed, if not i'll delete no worries 😅)

Anybody else experiencing this? Started about an hour ago.

I'm trans and I like programming. I'm interested in discussing topics like homebrew games and web1 sites. is there a community for that?

We've just spun up a brand new photo sharing instance, based on Vernissage. It's similar to pixelfed, but more focused on photography (as opposed to general image and short video sharing).

Signups are open. Currently, it mandates an email address, but you can stick whatever you like in there, and it will be automatically verified. Signups do require approval, but you can also generate invites that (I believe) bypass the requirement for approval.

Anyway, you can check it out at https://photos.blahaj.zone/

As a consequence, we will be taking down our pixelfed instance. We'll leave it up until the end of August to give people a chance to migrate, but honestly, it had really low activity in any case, so it shouldn't have too much of an impact on people.

Vernissage may also end up having low activity, but even if it does, I like it so much more than pixelfed, that we'll keep it running even if it's just me using it :)

Edit - Piefed has been updated and returned to service

~~Thanks to someone posting a piefed security exploit without a disclosure period, we've had to pull blahaj piefed down without warning until a fix is implemented. This could take around 24 hours or so.~~

Edit - We're back!

~~We've had an issue with our databases. One of our fast database servers ran out of space, and then the second fast server ran out of space whilst replicating to the first.~~

~~As a result, we have fallen over to our backup database server, which runs on spinny disks rather than SSDs. Spinny disks means that it's got plenty of space to spare, but it's not fast. The backup DB server is currently replicating to our two main servers to get things back up and running again, but whilst that's happening, all of our services are running slow.~~

~~The good news is, we'll be back up and running as if nothing happened because our backup server saved the day. The bad news is, it may take another 24 hours or so, because the backup server is reliable but not fast!~~

According to https://lestat.org/, the Blåhaj instance is down notably more often. What is happening or causing this?

Is it server costs? If so, I'm willing to contribute, but how should I do so? And if not, what would it be? Host issues? Targeted attacks, perhaps?

Over the last several days, I've noticed significant delays with votes, comments, and notifications from that instance. The longest delay I've noticed was 2 days. Is this a known issue? Is anyone else experiencing this?

Context

TL;DR This is a pledge to defederate from lemmy.world if they end up defederating from any of the instances in the Fediverse Anarchist Flotilla (FAF), which consists of https://lemmy.dbzer0.com/, https://anarchist.nexus/, and https://quokk.au/. Just want to see if the admins are aware of this situation, and see what others think.

With this version user badges are always shown next to usernames. There are also various bug fixes, and again security fixes

• Display UserBadges for Bot, Banned and Deleted users in all PersonListings by @MrKaplan-lw in #4035
• Increase timeouts for db pool by @nutomic in #6441
• Add private IP check for webmention by @nutomic in #6444
• Proper fix for nested comment fetch by @nutomic in #6451

Security

• Lemmy allows an authenticated low-privileged user to create a link post through POST /api/v3/post. When a post is created in a public community, the backend asynchronously sends a Webmention to the attacker-controlled link target. The submitted URL is checked for syntax and scheme, but the audited code path does not reject loopback, private, or link-local destinations before the Webmention request is issued. This lets a normal user trigger server-side HTTP requests toward internal services. https://github.com/LemmyNet/lemmy/security/advisories/GHSA-3jvj-v6w2-h948
• Lemmy fetches metadata for user-supplied post URLs and, under the default StoreLinkPreviews image mode, downloads the preview image through local pict-rs. While the top-level page URL is checked against internal IP ranges, the extracted og:image URL is not subject to the same restriction. As a result, an authenticated low-privileged user can submit an attacker-controlled public page whose Open Graph image points to an internal image endpoint. Lemmy will fetch that internal image server-side and store a local thumbnail that can then be served back to users. https://github.com/LemmyNet/lemmy/security/advisories/GHSA-h6hf-9846-xwrq

We've seen a lot of spam come from lemmy.org and we have therefore defederated from their instance. For the moment they have open registrations which only require filling out a simple captcha to create an account which many people abuse.

The admin is active but rarely takes moderation actions, but I've sent a DM to them with our concerns. We'll see how permanent this is.

try with any lemmy.blahaj.zone post, only lem.lemmy.blahaj.zone embeds.

I made an advice community for Swedish-speakers, and I happened to find one for Russian speakers. I'd mod, but I don't speak Russian.

It's called !advice_russian@lemmy.blahaj.zone

The moderator appears to have been banned

image to make my point. If I make an image post from blahaj (using the lem.lemmy.blahaj.zone app) it shows up as a link to an external source on the official app^[https://lem.lemmy.blahaj.zone/post/41085392], alexandrite^[https://alx.lemmy.blahaj.zone/lemmy.blahaj.zone/post/41085392] and blorp^[https://blorp.blahaj.zone/home/c/main%40lemmy.blahaj.zone/posts/https%3A%2F%2Flemmy.blahaj.zone%2Fpost%2F41085392], i. e. it doesn't embed. It looks normal everywhere else, i. e. from federated instances using the official web app^[https://lemmy.ml/post/45728815], and from the other web apps hosted locally^[https://phtn.lemmy.blahaj.zone/post/lemmy.blahaj.zone/41085392] ^[https://mlmym.lemmy.blahaj.zone/post/41085392] ^[https://tes.lemmy.blahaj.zone/post/lemmy.blahaj.zone/41085392].

Also alt text gets swallowed. It apparently doesn't federate, I can see it when I click on edit, the box is prefilled with "I am beanis cupcake and I will eat you" but this alt-text doesnt show up when looking at this post from a federated instance. It does show up on local frontends that implement this functionality.

Thank you for your attention to this matter.

Changes

This release addresses another security advisory related to internal host access. You can now bypass these checks for federation, in order to federate with instances over the local network by setting environment variable DANGER_FEDERATION_ALLOW_LOCAL_IP=1. There are also some bug fixes, and lemmy-ui now logs file requests.

• Improve IP checks by @nutomic in #6411
• Allow to bypass federation IP checks with env var DANGER_FEDERATION_ALLOW_LOCAL_IP by @nutomic in #158
• Fix Arabic user/community names by @nutomic in #3968
• Fix removing post.url by @nutomic in #3984
• Add lemmy-ui request logs by @MrKaplan-lw in #3933

Can blahaj commit to either blocking all of the ips on the tor exit list, or white listing them. This "block only a few" makes for a dice roll for tor users. Specifically the pictrs service just doesn't serve photos to banned ips but lemmy still works so it just partially breaks.

Asking for a friend.

This release mitigates a potential security issue with the image endpoint. In short, an attacker can inject query parameters and make proxied requests to arbitrary URLs. See the security advisory for details.

Also there are fixes for the database connection pool. The pool size is now at least two, as a lower size can result in deadlocks. Additionally there are now connection timeouts added. If your server logs show pool timeout errors, you should increase database.pool_size in the Lemmy config.

• Fix for image proxy filetypes by @dessalines in #6357
• Enable DB connection timeout by @Nutomic in #6355
• Use min database pool size of 2. by @dessalines in #6345

https://github.com/LemmyNet/lemmy/security/advisories/GHSA-jvxv-2jjp-jxc3

We’ve been seeing frequent spam waves of targeted harassment from accounts made on programming.dev over the course of several weeks, which has among other unacceptable things contained transphobia, obscene pictures of animals, calling for the victim to commit suicide, and other hateful rhetoric.

For these reasons we are defederating from programming.dev. We may refederate in the future if we have reason to believe there will be no further problems like these from their instance.

Edit: I've spoken with the admins there and I may refederate once I have more details on their application system. To be clear they don't support anything posted by these accounts and are deleting them as they are detected.

Edit: We've refedereted.

I cannot access !4tran4@sh.itjust.works from lemmy.blahaj.zone, a couple friends also had the same problem, other communities from sh.itjust.works are accessible though, and 4tran4 is accessible from other instances like lemmy.world

I'm only logged in on Voyager right now and there's nowhere where I can reset my password, and I can't log in on web to do that since I forgot my password. I didn't provide an email when registering, am I screwed or is there any hope? Worst case scenario, this might be my push to switch to the Piefed instance I guess

EDIT: I managed to log in on my other device by ripping my token out of Voyager, yes it's a very janky solution that probably won't last long but it works for now

EDIT 2: Turns out, I did have an email attached to this acc, just a really obscure one I never used. Problem solved 😁