How do you train an image style and speech recognition on your own machine?

Pictures and sound seem far from text, but the ideas of this series carry over: frozen weights with a small adapter, tests on what was held out, and a separate measure for what must not change. This article has two projects, both of which ran on an Apple M5 Max (PyTorch 2.15, Transformers 5.22). The timings below include other work running on the machine, so they run high. No NVIDIA card was tested, and neither project uses real people's images or voices.

Project A: teach a small image model a new style

A diffusion model learns to predict the noise added to an image, and then creates images by removing noise step by step. Full-size image models need gigabytes. To keep this in reach, 12a_image_style.py draws its own pictures at 32 by 32 pixels: a circle or a square, in style A (soft blue blobs on a pale ground) or style B (orange-and-black hatching on dark). It builds a network of 597,571 parameters, told which shape to draw.

The base model trains on 4,000 images of style A only. Then it meets 30 images of style B, through an adapter on its convolution layers:

class LoRAConv(nn.Module):
    """A frozen convolution plus a small trainable correction."""
    def __init__(self, conv, rank=4):
        super().__init__()
        self.conv = conv
        for p in conv.parameters():
            p.requires_grad_(False)
        self.down = nn.Conv2d(conv.in_channels, rank, conv.kernel_size, conv.stride, conv.padding, bias=False)
        self.up = nn.Conv2d(rank, conv.out_channels, 1, bias=False)
        nn.init.normal_(self.down.weight, std=0.02)
        nn.init.zeros_(self.up.weight)     # starts as "no change at all"

    def forward(self, x):
        return self.conv(x) + self.up(self.down(x))

This applies the LoRA idea to images. Zeroing up means the adapted model starts out identical to the base.

How do you grade a picture without a person looking at 200 of them? The script trains two small judges: one names the style, the other names the shape. Checked on real drawings, the style judge scored 100% on both styles, and the shape judge 100% and 99.7%. Then it asks for a mix of shapes, and judges 200 samples per model.

A chance level for shape is 50%. So the style moved completely in all cases, and it was the content that suffered, which is the image version of project ten. Measuring the style alone would have reported total success.

The second finding is less flattering for adapters. On this toy problem they gave no quality advantage. A model from scratch on the same 30 pictures did as well, because the task is so simple. What an adapter certainly gives you is size: rank 4 is about 5% of the model, and it can be switched on and off. On real image models, trained on far richer data, the case for reusing them is much stronger, but this project did not test that. Tutorials for running Stable Diffusion can start you there, with its own hardware needs and licence.

Project B: teach recognition, and test on strangers

The audio comes from the macOS say command, so the script works on a Mac. Linux and Windows users would need another voice generator, which was not tested. There are 27 English voices, 12 command words such as "open" and "cancel", three speaking rates: 972 clips. Each becomes a mel spectrogram, a picture of sound energy over time, fed to a small network of 24,300 parameters. Training took about 7 seconds. This kind of task, spotting a limited set of words, is what the Speech Commands dataset was built for, and its authors argue that it needs its own data rather than sentence recordings.

The 14-point drop between the first two rows is leakage again: a random split lets the model recognize the voice. The last row shows what to collect: more speakers, even with few clips from each. These are synthetic voices, so expect real microphones, rooms and accents to be harder.

Sentence recognition is too large a job to train from nothing here, so the second part adapts a ready model. Whisper tiny.en stumbles on names it has never heard. The script uses six invented ones, such as Zorvane and Kelmora. Only the decoder, the part that writes the words, was trained (29.6 million parameters): 300 steps, 323 seconds, from 576 clips by 12 voices. The test uses other sentences and other voices.

Word error rate counts the words substituted, added or dropped, per word spoken. The gain on names is large, and the cost is visible in the last row. The model got worse at everyday speech. One output after training read "Our partner Zorvane launched the Quillby" for "Our partner Zorvane wants a quote": the training names started appearing where they do not belong. If you only tested names, you would ship that. Before training, try Whisper's cheaper options, such as a larger model or giving it the tricky names up front.

Whose images and voices are you allowed to train on?

Nothing here used a real person's face, art or voice, and that kept the project simple. Your own project may not be so simple. This is general information, not legal advice.

A voice or face can identify a person, so treat these recordings as sensitive personal information. In Québec, biometric data has its own rules, so check with the person responsible for privacy before collecting any. Also check the base model's licence, because it follows your adapter. The clips generated here stay on your disk and are not published as a dataset.