Building on top of Rouslan Dimitrov's Microcontroller Class D Amplifier, I developed the idea to feed the power amplifier with PCM data directly. Rouslan does some dithering and noise shaping, but I'm convinced that proper interpolation would sound better.
The ATtiny is too slow to do even cubic interpolation so I'll move that into the host.
There are many fun peripherials that you can connect to a RPi but PC users are not left out either
* attiny85 has SPI and i2c interface so could be possibly used as USB<->i2c or USB<->SPI bridge. Unfortunately, the USB support needs some non-standard timings so there is quite a bit of integration and debugging needed.