That's not as much a difference as one might think because the flour too is mainly "sugar", specifically it's starch which gets turned into glucose same as the sweet tasting sugars (side note: it's quite an interesting process since saliva itself containes enzymes that break the starch into glucose and you can actually test this yourself using iodine solutions - which you should be able to get from a pharmacy - which turns starch purple).

Ultimatelly when it comes to nutrition, you should care mainly about carbohydrates in general (which includes all the stuff that is not sweet tasting but gets turned into glucose by the human body) rather than sugars specifically.

Once you look at it this way you'll find out that a ton of stuff which is not sweet is none the less rich in "sugars", namelly things like bread, pasta, polished rice and so on.