Also note that there's OnionShare, too. You don't need a TLS certificate or domain, don't need to port forward and can run it from home safely, routes over Tor so very hard to know you are even sharing something, well known and open source etc.

No, in that case there's no labelling required. That would be unsupervised learning.

https://en.wikipedia.org/wiki/Unsupervised_learning

Conceptually, unsupervised learning divides into the aspects of data, training, algorithm, and downstream applications. Typically, the dataset is harvested cheaply "in the wild", such as massive text corpus obtained by web crawling, with only minor filtering (such as Common Crawl). This compares favorably to supervised learning, where the dataset (such as the ImageNet1000) is typically constructed manually, which is much more expensive.

Ground truth labels are just prescriptive labels that we recognize as being true. The main thing that distinguishes unsupervised from supervised is that in unsupervised learning, what is "good" is learned from the unstructured data itself. In supervised learning, what is "good" is learned from some external input, like "good" human-provided examples.

No, it's unsupervised. In pre-training, the text data isn't structured at all. It's books, documents, online sources, all put together.

Supervised learning uses data with "ground truth" labels.

This pre-training was done by Meta. It's what Llama-3.1-405B is (in contrast to Llama-3.1-405B-Instruct). https://huggingface.co/meta-llama/Llama-3.1-405B

Training Data

Overview: Llama 3.1 was pretrained on ~15 trillion tokens of data from publicly available sources. The fine-tuning data includes publicly available instruction datasets, as well as over 25M synthetically generated examples.

Unsupervised training happens during the pre-training phase when you dump all kinds of quality documents and it learns the relationship between tokens

Curious, how do you evaluate the performance without breaking the law?

The article you linked to uses SFT (supervised fine tuning, a specific training technique) as its alignment strategy. There are other ways to fine-tune a model.

I guess I'm wondering if you can train on these partial responses without needing the full rest of the output, without the stop token, or if you need full examples as the article hints to.

You are right. Their description of "SOTA Open Source TTS" caused me to assume it was open source, but it's clear that

This codebase and all models are released under CC-BY-NC-SA-4.0 License.

So, it's "source available" and not released under a permissive licence.

Thank you so much, that exactly answers my question with the official response (that guy works at Meta) that confirms it's the same base model!

I was concerned primarily because in the release notes it strangely didn't mention it anywhere, and I thought it would have been important enough to mention.

I followed their instructions here: https://speech.fish.audio/

I am using the locally-run API server to do inference: https://speech.fish.audio/inference/#http-api-inference

I don't know about other ways. To be clear, this is not (necessarily) an LLM, it's just for speech synthesis, so you don't run it on ollama. That said I think it does technically use Llama under the hood since there are two models, one for encoding text and the other for decoding to audio. Honestly the paper is terrible but it explains the architecture somewhat: https://arxiv.org/pdf/2411.01156

On Lemmy, everything is a bit leftist at the moment.