4
submitted 7 months ago* (last edited 7 months ago) by BluePower@mastodon.social to c/perchance@lemmy.world

[Bug] Downloading revisions not working

I made a similar post about this one (https://lemmy.world/post/11134086) and it happened again, and hopefully the Dev can notice. (ping please)

Here's the error that Perchance throws when I try to download revisions from https://perchance.org/power-generator-manager-10k-milestone:

Uncaught URIError: URI malformed
at decodeURI (see image)

EDIT: After some later investigation, I've found out that this does NOT affect earlier revisions (see this comment: https://lemmy.world/comment/8770044)

@perchance

you are viewing a single comment's thread
view the rest of the comments
[-] perchance@lemmy.world 4 points 7 months ago

Hmm I think this may have been caused by multiple concurrent save requests, which added multiple versions of the same "diff patches" to the revisions list, or put them out of order, or something. Not entirely sure. I did fix the "save traffic jam" problem a few weeks ago like I mentioned to you in another post, and I just added some server-side checks to make sure that a save request is rejected if there's already one in progress. Hopefully that fixed it, but it's possible that there's another issue.

Unfortunately, regardless of this fix, your current revisions are corrupted (it's likely possible to fix with some manual work, but it'd not be easy). If it's particularly important then it is possible for you to access the raw diff data by opening the browser console and typing window.diffStuff.outputTemplatePatches but it'd require some expertise and probably an hour or two of work.

Assuming the revisions of that generator aren't super important to you, the best way to "fix" it is to duplicate the generator, then delete the old one, and claim the old name with the new one. And please let me know if it happens again - that way I can look further into this, knowing that it's not due to concurrent save requests.

Sorry about that!

[-] BluePower@sh.itjust.works 3 points 7 months ago* (last edited 7 months ago)

Update regarding on the revisions: The bug seems to affect later revisions, and not the entire revision list. And here's what I've done so far.

When trying to download some earlier revisions in power-generator-manager, I have to say that revisions only made after 12 January 2024, 23:31:11 UTC+7 (after index 863) are affected by this bug. I even ran this line of code in the console to get a decoded URI of the latest model patch:

decodeURI(diffStuff._patchesToRevision(diffStuff.modelTextPatches, 1617))

And that returned the exact same error, URI malformed. So that's a problem with the encoded text on each specific revision.

But my investigation doesn't stop there. To see if there's a single character that triggers the entire URI corruption, I also compared those raw revision data of index 863 and 864, decoded manually, in a text comparison app, and I've finally found the culprit, in line 204:

Revision Sat Jan 13 2024 13:09:52 GMT+0700 (GMT+07:00):

<li><b>The generator manager is getting another big update!</b> ... Scroll down past the end of the announcement to see the full changelog of the update!</li>

Revision Fri Jan 12 2024 23:31:11 GMT+0700 (GMT+07:00):

<li><b>The generator manager is getting another big update!</b> ... Scroll down past20the announcement to see the full changelog of the%2!</li>

And there it is, the %2 that triggered the URI malformed problem. There was apparently some sort of a glitch with the diffStuff._patchesToRevision() function when it tries to parse the raw patch data into a revision (that could be on the concurrent save requests you've mentioned).

I then tried the same procedure in the latest revision of the hub page and found that the %2 character started spreading around the generator code multiple times. Not only that, but parts of the entire data are also scrambled pretty bad, too. (This doesn't happen on unaffected revisions!) So your statement on "your current revisions are corrupted" is kind of true.

And now, every time I had to get a revision, I'd have to get the patch directly with that specific approach, manually decode it, transfer it into a notepad, and then saving it if I wanted to (unless there's a way to fix that URI corruption thing).

And yes, I did check the outputTemplatePatches and everything in it is okay, no corruption whatsoever. Even to the latest one ๐Ÿ™‚

[-] perchance@lemmy.world 3 points 7 months ago* (last edited 7 months ago)

Great work on this investigation! If you can manage to reproduce this starting with a fresh generator, that would be extremely helpful. I.e. if you can find some procedure of edits, starting from perchance.org/blank which causes revisions to get corrupted, I'd be able to solve the problem very quickly.

now, every time I had to get a revision, Iโ€™d have to get the patch [...]

Why not fork it and start with a fresh? That will reset your revisions. And you can of course just rename the old one so you can claim the old name with the new gen. I'm hoping that this was just an issue with concurrent saves, in which case it shouldn't happen again. Or maybe with the above quote you mean that's what you'd have to do if you didn't fork the generator.

[-] BluePower@sh.itjust.works 3 points 7 months ago* (last edited 7 months ago)

Thanks! I will try to find more generators affected by this bug and maybe start messing around with that blank page (revisions worked on 2 of my easter egg generators so far). But for now, here's a little snippet that I would use to check if a generator's revision is affected by the bug:

for (let i = 0; i < diffStuff.modelTextPatches.length; i++) {
  decodeURI(diffStuff._patchesToRevision(diffStuff.modelTextPatches, i));
  // when executed, the console tries to evaluate that so that's why it'll get an error if the revision's corrupted
  console.log("modelText: " + i);
  // count COUNT COUNT
}
for (let i = 0; i < diffStuff.outputTemplatePatches.length; i++) {
  decodeURI(diffStuff._patchesToRevision(diffStuff.outputTemplatePatches, i));
  // same with this too ^^
  console.log("outputTemp: " + i);
}
// you'd have to load the revisions in order for it to work, and for generators with lots of revisions, it'll probably take longer to finish the check

Which in this case, I just ran on my milestone announcement page, and it returned this:

(Also @VioneT@lemmy.world, @Alllo@lemmy.world, @eatham@aussie.zone and others, if you wanted to contribute by finding some buggy revisions in your generators by running that in the console and found errors like that, that'll be very helpful.)

Why not fork it and start with a fresh? That will reset your revisions.

Seems like you have a good point. ๐Ÿ™‚ I will try this with my milestone announcement page after my preparation with it is fully done.

[-] Alllo@lemmy.world 3 points 7 months ago

i basically have no idea any of this but will look at what all this is and do it tonight because you say it will be helpful (tonight here is like 5 hours from now)

[-] Alllo@lemmy.world 2 points 7 months ago* (last edited 7 months ago)

@BluePower@sh.itjust.works I just went through every one of my generators and got the latest revision without issue.

also i wasn't able to reproduce the error on perchance.org/blank

i guess i can't help that i know of. sorry :)

at least im making cool things like i hope u are too!

[-] BluePower@sh.itjust.works 2 points 7 months ago

Nice! At least you're trying to :)

[-] VioneT@lemmy.world 1 points 7 months ago* (last edited 7 months ago)

Here is probably a smaller code to check, also it wouldn't stop at the first malformed:

let text;
let x = diffStuff.modelTextPatches; // Change to `outputTemplatePatches`
x.forEach((a,i) => {
  try {
    text = decodeURI(diffStuff._patchesToRevision(x, i));
    console.log('Checked: ', i);
    text = '';
  } catch (err) {
    console.error('Error: ', i, err);
  }
})
console.log('Check Complete');

You can also just copy and paste it on the console directly. Just need to load revisions first.

It seems the /hub has the last '101' revisions on modelTextPatches to be malformed. The outputTemplatePatches didn't throw any errors. Looking at the other pages that I have, mostly previous revisions are malformed. The date it started appearing (at least on my revisions on ai-generated-realistic-portraits) is Jan 8, 2024 about 10:38:43 AM UTC, but about 10:38:34 AM the revisions are still okay. On the hub it is also Jan 8, 2024 about 11:30:42 AM UTC.

EDIT: It seems to be only generators that were updated on Jan 8 before 12:03:58 PM and after 10:38:34 AM seems to have problems on my end.

[-] BluePower@sh.itjust.works 1 points 7 months ago* (last edited 7 months ago)

That's interesting. I didn't know the hub also has the same revisions problem. I've been tinkering with that code for hours and this is what I made:

function checkForMalformedRevs(patch, loglabel, startFrom) {
  let text;
  let dateRegex = [/^.* ([0-9][0-9]:[0-9][0-9]:[0-9][0-9]) GMT+.*/g, "$1"];
  console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Started.');
  patch.forEach((a,i) => {
    if (i < startFrom) return;
    let pi = diffStuff._patchesToRevision(patch, i);
    let x = 0;
    let offset = 0;
    try {
      for (x = 0; x < pi.length; x += 100) {
        offset = (!!pi.substring(x, x + 100).match(/%.$/g) * 1) + (!!pi.substring(x, x + 100).match(/%$/g) * 2); // checks if there's any "chopped" URI characters at the end of the chunk to prevent `URI malformed` errors just because of that
        text = decodeURI(pi.substring(x, x + 100 + offset));
        text = '';
        x += offset;
        offset = 0;
      }
      console.log('[' + loglabel + ']', Date().replace(...dateRegex), 'Checked:', i);
      x = 0;
    } catch (err) {
      console.error('[' + loglabel + ']', Date().replace(...dateRegex), 'Error:', i, 'at chunk', x + '-' + (x + 100 + offset), 'of', pi.length, err);
    }
  })
  console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Complete.');
}
checkForMalformedRevs(diffStuff.modelTextPatches, "modelTextPatches");
checkForMalformedRevs(diffStuff.outputTemplatePatches, "outputTemplatePatches");

There we go! It's a greatly enhanced version of the checker tool. To reduce all the wait, the checker first processes a chunk of data, then if that chunk of data has been checked successfully, then the next chunk will be checked. (But still, if the revision data has around tens of thousands of characters it's going to take seconds to process)

The process roughly looks like this:

"generatorStats%20=%20%7Bimport:generator-stats-plugin%7D%0%0Atitle%0A%20%20Power%20Generator%20Manager%0A%0A%0A%0A\n\n"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < checking

The checker will take a part of the data and then decodeURI's them. Then if it's successful, another chunk is also checked:

"generatorStats%20=%20%7Bimport:generator-stats-plugin%7D%0%0Atitle%0A%20%20Power%20Generator%20Manager%0A%0A%0A%0A\n\n"
                                                  ~~~~~~~~!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < checking

And there, once it's URI malformed time, the checker stops for checking further chunks. Additionally, the logger not only throws the error, but also locates the chunk location where the error is throwed and shows when the check happened in time. You can also set on which index the checker should start checking with startFrom for easier debugging.

Additional note: While testing, I also found that other "suspicious" (invalid) characters (including %9F) and the single percentage % are also affecting this whole revisions problem.

(Also I might create a whole "utility" generator out of this so more people can contribute to this ๐Ÿ˜Š)

[-] VioneT@lemmy.world 1 points 7 months ago* (last edited 7 months ago)

On that note, I duplicated the hub and deleted the old one, it did resolve the revisions problem (though the view count reset).

[-] BluePower@sh.itjust.works 1 points 7 months ago

What? With a hub that had over a thousand views? That's remarkable.

Also @perchance@lemmy.world, if there's a way to backup and erase all the revisions somehow ๐Ÿค”

[-] VioneT@lemmy.world 2 points 7 months ago* (last edited 7 months ago)
function checkForMalformedRevs(patch, loglabel, startFrom) {
  let text;
  let errors = []
  let dateRegex = [/^.* ([0-9][0-9]:[0-9][0-9]:[0-9][0-9]) GMT+.*/g, "$1"];
  console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Started.');
  patch.forEach((a,i) => {
    if (i < startFrom) return;
    let pi = diffStuff._patchesToRevision(patch, i);
    let x = 0;
    let offset = 0;
    try {
      for (x = 0; x < pi.length; x += 100) {
        offset = (!!pi.substring(x, x + 100).match(/%.$/g) * 1) + (!!pi.substring(x, x + 100).match(/%$/g) * 2); // checks if there's any "chopped" URI characters at the end of the chunk to prevent `URI malformed` errors just because of that
        text = pi.substring(x, x + 100 + offset)
        let decoded = decodeURI(text);
        decoded = '';
        text = '';
        x += offset;
        offset = 0;
      }
      console.log('[' + loglabel + ']', Date().replace(...dateRegex), 'Checked:', i);
      x = 0;
    } catch (err) {
      console.error('[' + loglabel + ']', Date().replace(...dateRegex), 'Error:', i, 'at chunk', x + '-' + (x + 100 + offset), 'of', pi.length, err, 'Chunk Text:', text);
      errors.push({date: new Date(diffStuff.patches[i].creationTime), log: loglabel, revisionId: i, chunkRange: `${x}-${x + 100 + offset}`, revisionLength: pi.length, text: text});
      
    }
  })
  console.log('[' + loglabel + ']', Date().replace(/ \(GMT+.*\)/g, ""), 'Check Complete.');
  console.table(errors, ['date', 'text']);
}
checkForMalformedRevs(diffStuff.modelTextPatches, 'modelTextPatches')
// checkForMalformedRevs(diffStuff.outputTemplatePatches, 'outputTemplatePatches')

Here is a code that also outputs the Date of the Revision as well as the Text that was throwing the error neatly in a table. It seems ONLY the modelTextPatches are throwing the malformeds.

[-] BluePower@sh.itjust.works 1 points 7 months ago

Oooh nice! I'll test that later, and I'll improve it even more when needed.

[-] BluePower@sh.itjust.works 2 points 7 months ago* (last edited 7 months ago)

Thanks for the explanation and the possible fix! That would've been sucked if my revisions are possibly messed up - especially if I had to download some just to revisit some of my older versions of that generator, especially the power-generator-manager one.

I also want to note, for the new save, whenever the save button get stuck on the disabled "saving" state, I'd just reload the generator using the "reload" button (and maybe check in the DevTools network just to make sure) to ensure that the save process is fully done. (Might also create a separate post for this)

this post was submitted on 24 Mar 2024
4 points (100.0% liked)

Perchance - Create a Random Text Generator

449 readers
11 users here now

โš„๏ธŽ Perchance

This is a Lemmy Community for perchance.org, a platform for sharing and creating random text generators.

Feel free to ask for help, share your generators, and start friendly discussions at your leisure :)

This community is mainly for discussions between those who are building generators. For discussions about using generators, especially the popular AI ones, the community-led Casual Perchance forum is likely a more appropriate venue.

See this post for the Complete Guide to Posting Here on the Community!

Rules

1. Please follow the Lemmy.World instance rules.

2. Be kind and friendly.

  • Please be kind to others on this community (and also in general), and remember that for many people Perchance is their first experience with coding. We have members for whom English is not their first language, so please be take that into account too :)

3. Be thankful to those who try to help you.

  • If you ask a question and someone has made a effort to help you out, please remember to be thankful! Even if they don't manage to help you solve your problem - remember that they're spending time out of their day to try to help a stranger :)

4. Only post about stuff related to perchance.

  • Please only post about perchance related stuff like generators on it, bugs, and the site.

5. Refrain from requesting Prompts for the AI Tools.

  • We would like to ask to refrain from posting here needing help specifically with prompting/achieving certain results with the AI plugins (text-to-image-plugin and ai-text-plugin) e.g. "What is the good prompt for X?", "How to achieve X with Y generator?"
  • See Perchance AI FAQ for FAQ about the AI tools.
  • You can ask for help with prompting at the 'sister' community Casual Perchance, which is for more casual discussions.
  • We will still be helping/answering questions about the plugins as long as it is related to building generators with them.

6. Search through the Community Before Posting.

  • Please Search through the Community Posts here (and on Reddit) before posting to see if what you will post has similar post/already been posted.

founded 1 year ago
MODERATORS