858

A purported leak of 2,500 pages of internal documentation from Google sheds light on how Search, the most powerful arbiter of the internet, operates.

The leaked documents touch on topics like what kind of data Google collects and uses, which sites Google elevates for sensitive topics like elections, how Google handles small websites, and more. Some information in the documents appears to be in conflict with public statements by Google representatives, according to Fishkin and King.

top 50 comments
sorted by: hot top controversial new old
[-] woelkchen@lemmy.world 358 points 3 months ago

Some information in the documents appears to be in conflict with public statements by Google representatives

I would have never guessed that.

[-] applepie@kbin.social 111 points 3 months ago

At this point if you are not assuming that corporation is pretty much lying for convenience. you aint operating in reality haha

[-] trolololol@lemmy.world 46 points 3 months ago

Yep but I'll add my two cents, half is lying and half is guessfull ignorance because nobody really knows how big and old systems really work.

load more comments (2 replies)
[-] CosmoNova@lemmy.world 92 points 3 months ago

Crazy how self regulation always winds up like this. By crazy I mean predictable of course.

[-] iopq@lemmy.world 12 points 3 months ago

You're supposed to move to a different search engine for the market to work. I already have, have you?

[-] NeatNit@discuss.tchncs.de 11 points 3 months ago

This approach is doomed to fail, so long as the general public isn't aware of the problem or its scale. Government regulation is the only way.

load more comments (1 replies)
load more comments (3 replies)
[-] deweydecibel@lemmy.world 10 points 3 months ago

This doesn't have anything to with regulation. This is mainly a bunch of SEO and marketing people whining that Google hasn't been honest with them in telling them exactly how to game their search engine.

[-] cm0002@lemmy.world 23 points 3 months ago

Well I am just shocked, SHOCKED. Well, not that shocked.

load more comments (2 replies)
[-] flappy@lemm.ee 92 points 3 months ago

Can't wait for selfhosted web search to become better.

[-] jonne@infosec.pub 61 points 3 months ago

You mean hosting your own crawler/indexer? That doesn't really sound like a thing you could do cost-effectively.

[-] zutto@lemmy.fedi.zutto.fi 19 points 3 months ago

Surprisingly, it's very doable, requires basic technical knowledge and relatively minimal computing resources (runs in the background on your computer).

https://yacy.net/ Github

I have tampermonkey script that sends yacy to crawl any websites that I visit, and it's keeping up relatively good index for personal use of the visited websites. Combine yacy with ~300gb of Kiwix databases, add searxng as a frontend and you have pretty strong self hosted search engine.

Of course you need to supplement your searches from other search engines, as yacy does not crawl the whole web, just what you tell it to.

I encourage anyone who's even slightly interested on this stuff to try Yacy, it's ancient piece of software, but it still works very well and is not an abandoned project yet!

--

I personally use Yacy mostly on private mode, but it does have the distributed network there as well. Yacy current freeworld status

load more comments (3 replies)
[-] warmaster@lemmy.world 16 points 3 months ago
[-] Paradox@lemdro.id 51 points 3 months ago

Federated directories. We're going back to Yahoo like it's 1995

[-] NoIWontPickAName@kbin.earth 32 points 3 months ago
[-] AbidanYre@lemmy.world 19 points 3 months ago

<under_construction.gif>

load more comments (4 replies)
load more comments (4 replies)
[-] brbposting@sh.itjust.works 16 points 3 months ago

Right!

Before his company was able to block more of Microsoft's own tracking scripts, DuckDuckGo CEO and founder Gabriel Weinberg explained in a Reddit reply why firms like his weren't going the full DIY route:

“… [W]e source most of our traditional links and images privately from Bing … Really only two companies (Google and Microsoft) have a high-quality global web link index (because I believe it costs upwards of a billion dollars a year to do), and so literally every other global search engine needs to bootstrap with one or both of them to provide a mainstream search product. The same is true for maps btw -- only the biggest companies can similarly afford to put satellites up and send ground cars to take streetview pictures of every neighborhood.”

Ars

load more comments (2 replies)
[-] Jako301@feddit.de 18 points 3 months ago

How is that even supposed to work? These search engines need per definition massive databanks to search through. Either you need your own crawler and indexer which is more than just inefficient, or you are limited to a relatively short list of curated static results.

[-] Peppycito@sh.itjust.works 12 points 3 months ago

If they're taking tips from Google, why would they get better?

[-] FaceDeer@fedia.io 32 points 3 months ago* (last edited 3 months ago)

Google actually was good, so there's probably some good information in this documentation. If nothing else we can perhaps figure out what "went wrong."

Edit: I've been reading the blog post that appears to be the main person the leak was shared with and there's a lot of in-depth analysis being done there, but I'm not seeing a link to the actual documents. This is a huge article, though, I might be overlooking it.

load more comments (1 replies)
load more comments (7 replies)
[-] deweydecibel@lemmy.world 83 points 3 months ago* (last edited 3 months ago)

Rand Fishkin, who worked in SEO for more than a decade, says a source shared 2,500 pages of documents with him with the hopes that reporting on the leak would counter the “lies” that Google employees had shared about how the search algorithm works.

Am I supposed to care that the poor SEO assholes that need to get their ads more visibility weren't being given all the instructions on how to do that by the search engine?

Most of this article is SEO "experts" complaining that some of the guidelines they were given didn't match what's in the internal documents.

Google is shit, but SEO is a cancer too. I can't be too bothered by Google jacking them around a bit.

[-] lvxferre@mander.xyz 31 points 3 months ago* (last edited 3 months ago)

And I supposed to care that the poor SEO assholes that need to get their ads more visibility weren’t being given all the instructions on how to do that by the search engine?

No. You're supposed to care that a company is pointlessly* lying, thus it's extremely likely to deceive, mislead and lie when it gets some benefit out of it.

In other words: SEO arseholes can ligma, Google is lying to you and me too.

*I say "pointlessly" because not disclosing info would achieve practically the same result as lying.

[-] brbposting@sh.itjust.works 24 points 3 months ago

need to get their ads more visibility

I occasionally encounter the desire for a search engine to surface non-advertisement content :)

Now if they lied to advertisers and told small bloggers, reputable news agencies, fediverse admins, etc. the insider secrets… now we’re talkin’!

[-] WindyRebel@lemmy.world 8 points 3 months ago* (last edited 3 months ago)

Edit: If you’re going to downvote me, please take the time to explain why you think I’m wrong. Stop being the hive mind.

Tell me you don’t know shit about SEO without telling me you don’t know shit about SEO.

Just because there are people who do bad things doesn’t mean the industry is bad or have bad intentions. SEO isn’t ads. Advertorials can be a tactic of SEO, but it’s not SEO as a whole. Same with clickbait because it works, and I guarantee you also fall for it constantly.

SEO is about understanding what someone needs and creating an experience to ensure that someone finds the answer to what they need through content and/or a product to solve their needs.

This can be achieved through copywriting, researching search trends and queries, technical analysis of websites and how they render, providing guidance on helpful assets (photos, pdfs, videos, form, copy, etc), PR outreach because links are how people move around online or discover things, social planning because social media are a form of search engines, and more.

And finally, SEOs are not responsible for how Google treats shit. That’s Google who is responsible. Google is the one that tweaks the algorithm and doesn’t catch spammy shit. In fact many SEOs catch it and report it to Google’s reps, but they are the ones who can ensure the right team(s) fix the issue.

[-] Olap@lemmy.world 17 points 3 months ago

Fuck SEOs - that is why you are getting downvoted. Organic content creation has been ruined by you AND google. Own your problems, beg forgiveness, stop playing the stupid game where there are no winners

load more comments (9 replies)
load more comments (2 replies)
[-] frezik@midwest.social 8 points 3 months ago

Historically, Google had a give-and-take with SEO. You can't make SEO companies go away, but you can curb the worst behavior. Google used to punish bad behavior with a poor listing, and you had to do some work to get it back into compliance and tell Google it's fixed up.

It wasn't ideal, but it functioned well enough.

The drive to make search more profitable over the past few years seems to have meant dropping this. SEO companies can get away with whatever. If they now have the whole manual, game over. Google of a decade ago might have done something about it. Google of today won't bother.

[-] mlg@lemmy.world 81 points 3 months ago

Google has been pretty crap for a decade now.

I still remember demoing how easily they can manipulate people by searching "Pakistan News" and the results being exclusively all Indian media outlet propaganda way back in 2016.

I really feel like they never got properly exposed for this just because it's a search engine and not a social media, so people didn't care enough about it. Also because Google was still top of the game in most results compared to other sites back then.

[-] NutWrench@lemmy.world 68 points 3 months ago

Here's the sooper-secret search result algorithm for whatever you type into Google:

YouTube results, followed by Reddit results, followed by "Sponsored" results, followed by AI-written Bot results, then a couple pages of Amazon results and finally, on page 10 or so, a ten-year-old result that's probably no longer relevant.

[-] ipkpjersi@lemmy.ml 10 points 3 months ago

That's generally what I've found to be the case, shocking that it's considered so secret lol

[-] pyre@lemmy.world 65 points 3 months ago

awesome, now we can make our own search engine that is filled with complete trash and isn't concerned with helping the user at all.

[-] Danterious@lemmy.dbzer0.com 49 points 3 months ago* (last edited 3 months ago)

I guess we are going to be in for more SEO spam than usual if this document is accurate. But I think its good that we are finally going to get a better understanding how Google manipulates people with the algorithm.

So a win-lose situation.

~Anti~ ~Commercial-AI~ ~license~ ~(CC~ ~BY-NC-SA~ ~4.0)~

load more comments (1 replies)
[-] iopq@lemmy.world 48 points 3 months ago

I want a federated social bookmarking site. Not for news or discussion of recent stuff, but to keep some good sites in your account and to share with others.

Searching those and getting results with attached upvotes/downvotes would be ideal

[-] nucleative@lemmy.world 9 points 3 months ago

Interesting concept. Like if you could upvote/downvoted the SERP and it actually mattered and wasn't easy to manipulate.

load more comments (3 replies)
[-] nick@midwest.social 42 points 3 months ago

Now, where to download these, for science.

[-] Tronn4@lemmy.world 25 points 3 months ago

Use Bing 😅😅

[-] NoIWontPickAName@kbin.earth 18 points 3 months ago
[-] b3an@lemmy.world 32 points 3 months ago

Honestly I hope this bites them hard. They've done way way worse to small businesses and competition for decades now.

[-] dojan@lemmy.world 22 points 3 months ago

It's honestly quite strange that this sort of black box system is allowed to exist. How are governments around the world OK with a vast majority of the internet being filtered through a private company's lens without any sort of insight into how it works? That sounds skeevy as shit.

load more comments (14 replies)
[-] postmateDumbass@lemmy.world 18 points 3 months ago

I tried to cry for them but after Googling instructions about how to I poured Elmer's Wood Glue on both eyes. I cannot call the result tears. Not sure what to call it, but certainly not tears.

[-] AllNewTypeFace@leminal.space 14 points 3 months ago

Does Google still have a search algorithm? I thought they now just feed everything into a huge LLM and let it regurgitate statistically plausible answers.

[-] Mereo@lemmy.ca 20 points 3 months ago

Google Search uses a regular search algorithm. Google AI overview will be a product that feeds from Google search.

[-] SharkAttak@kbin.social 13 points 3 months ago

"How does Google's search algorithm works? -reddit"
The algorithm works with the use of proprietary tchnology.. (read 2500 pages more)

[-] autotldr@lemmings.world 12 points 3 months ago

This is the best summary I could come up with:


But how exactly Google ranks websites has long been a mystery, pieced together by journalists, researchers, and people working in search engine optimization.

Now, an explosive leak that purports to show thousands of pages of internal documents appears to offer an unprecedented look under the hood of how Search works — and suggests that Google hasn’t been entirely truthful about it for years.

“While I don’t necessarily fault Google’s public representatives for protecting their proprietary information, I do take issue with their efforts to actively discredit people in the marketing, tech, and journalism worlds who have presented reproducible discoveries.”

Fishkin told The Verge in an email that the company has not disputed the veracity of the leak, but that an employee asked him to change some language in the post regarding how an event was characterized.

The pervasive, often annoying tactics have led to a general narrative that Google Search results are getting worse, crowded with junk that website operators feel required to produce to have their sites seen.

The US government’s antitrust case against Google — which revolves around Search — has also led to internal documentation becoming public, offering further insights into how the company’s main product works.


The original article contains 906 words, the summary contains 200 words. Saved 78%. I'm a bot and I'm open source!

[-] Fedizen@lemmy.world 8 points 3 months ago

rubs hands together

load more comments
view more: next ›
this post was submitted on 28 May 2024
858 points (100.0% liked)

Technology

58160 readers
3011 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS