Introduction
Many synthesizers are powerful (and complicated) “all in one” units that can do almost anything. I wanted to test the opposite where several units (one for each instrument) work together as an orchestra.
Leet AI is a concept for a miniature synthesizer with the following design goals:
- Cute, small, playful and cheap (so you can have one for each instrument)
- Open source and customizable (programmed in python with COTS modules)
- Easy to use (the display shows current mode and options – no need for strange key combinations)
- Wireless communication and battery powered
- AI enabled (so it can generate new melodies for inspiration)
Please note that Leet AI is a concept that is missing lots of features, but it is mature enough to demonstrate the overall idea. If it gets good feedback, I will probably continue to work on it, and since it is open source – anyone can contribute :)
The synth is medium difficult to build – you need to solder SMD components and have access to a 3D printer.
Spec:
- 86 x 60 x 15 mm (smaller than a deck of cards)
- 16 keys with RGB LEDs
- powerful microprocessor running circuit python (ESP32-S2)
- 1.8” color display (160×128 pixels)
- 1000mAh rechargeable LiPo battery
- high quality 112dB SNR DAC
- low latency wireless communication (ESP-now)
- MIDI in/out over USB-C
- 2 rotational magnetic encoders with dual tilt navigation
Design process:
Leet AI is combining the best parts of three other projects – leet, leet modular and ChipChamp. When I played with those projects, I learned a lot and gained enough confidence to take the next step. The synth is based on circuit python, making it easy to modify and write new functions. When connected to a computer (over USB-C) the synth appears as an USB drive (allowing you to modify the code directly).
AI muse
I thought it would be nice if the synth could use AI to propose different melodies, tracks and drum patterns as inspiration. Instead of doing inference on the device – it connects (wirelessly) to a server that does the heavy lifting. The server runs a modified version of the getMusic project from Microsoft research – a diffusion model that that can generate new tracks to an existing midi file. It works quite well but like the early image generation diffusion models it requires some patience and most results are so-so, but every now and then something beautiful emerges and you get really amazed. This is still an early stage of generative AI and I’m convinced that the result will improve with better models in the future (if you know your stuff – you can even train it on your favorite music :)
The demo
I asked chatGPT write a chord progression program in python, based on a tunable Markov chain. The generated midi progression is used for the bass track. Another program composed a simple arpeggio that followed the progression and those two tracks were repeated six times. I used a midi editor to tweak the bass and removed the arpeggio in the intro and bridge. The AI was modified so it composed 20 versions of the lead and drum track. 15 versions were crap, 3 ok and 2 were great. I combined the two and Robert is your mother’s brother!
Since the audio generation has more to give – I rendered a final version of the song in Ableton. It needs more love (and skill) but is good enough to show the potential… The video shows how one unit acts as a conductor (keeping the other units in sync, outputting MIDI and rendering audio), while the other units correspond to each instrument, bass, lead, piano and drum.
Prototype
For the synth I started by adding a I2S DAC to a ChipChamp and experimented with different python libraries until it could play music. I combined the audiocore library for drum samples with synthio for wavetable synthesis.
I spent a lot of time tuning the graphics to make the piano roll animation. There was not enough processing power to move around the sprites, so I created a static grid where the sprites are turned on and off depending on the time in the melody. This works quite ok, but it would be nice with an optimized library with smother animation and where the height of the sprites is depending on the note duration…
Another rabbit hole was the wireless communication and time synchronization between the units. A lightweight custom protocol is used with of ESPnow (sending single Wi-Fi packets). Commands are broadcasted to all units and some packets are addressed to a certain instrument. When a device is powered on it sends a pair request to the conductor that then transmits all events for the instrument in the song.
Design
Once I got the basic functions working, I set the design goals and did a lot of sketches in Rhino with different components to find the right form factor.
I ended up with a single octave design with a display and two encoders for navigation and tuning. I think it looks like a cute version of a QY70.
A two-octave version with a wide OLED screen was considered, but I like the idea of stacking several units to cover several octaves. I used Inkscape to evaluate different design ideas and then completed the final 3D-model in Rhino.
Custom encoders
To keep it compact I used a traditional PCB instead of the 3DPCB. I did the schematics in KiCad and routed a first version with 12 keys (one octave). I didn’t want any protruding parts so I replaced the traditional mechanical rotation encoder with a magnetic version, but I also complemented the rotation with two keys so it could be used for menu navigation (next, prev, select and back). A 3D printed carrier holds a bearing in place and the flexible arms allows the bearing to tilt, pressing the upper or lower switches. I lost count of the number of iterations I went thru, but eventually it worked just as I had hoped for.
Since the AS5600 sensor have a fixed i2c address, I used the analog mode so I only needed a single I/Os for the position. When I briefly studied the datasheet, I thought I could disable the 10-LSB hysteresis in the configuration register, but later realized that the register only affects the digital mode, not the analog. This meant that there is a gap in the beginning and end of each rotation, which was quite annoying.
The modules for the LiPo charger and the magnetic encoders were too big to fit, so I replicated their design on the PCB. Instead of ordering the components separately, I desolder and resolder the components from the modules.
Second version – more keys
I planned to make two versions of the synth, one with a single octave layout, and one with a 16 step sequencer. After building the first version I realized that if I staggered 16 keys it would cover both needs. It’s one of those ideas that is so simple that it’s embarrassing that I didn’t think of it immediately…
I made a second version with 16 keys and replaced the analog encoder output with dual bit banged i2c interfaces. Since this version is slightly wider, I could fit the charger module inside making it easier to build. The synth is slightly smaller than a deck of cards and I think I like the proportions of the second version even more.
Sound generation
The synth is using the excellent synthio library for audio output. Each synth can use semi modular synthesis, samples or wavetables. When the Boss unit is rendering all the instruments, it is limited to fixed sample playback (for drums) and wavetable generation due to performance. To be honest, the generated sound is ok to play around with melodies, but nowhere near the fidelity required for production. If I learn how to write my own python libraries in the future, an optimized audio library that can mix different samples with different pitch would be powerful.
AI assisted development
Since the synths appears as a USB drive when connected to a computer, it is super easy to use AI code assistants like cursor.sh or continue.dev. Simply select the synth in the editor and tell the agent what you would like to develop. I used it a lot to improve the code structure and error handling. It is easy, powerful, and fun (especially when it occasionally goes off the rails :)
Future development:
There are tons of things to implement. I don’t recommend building it at this stage since almost everything is hard-coded for the demo and the unit is quite useless without implementing:
- Optimized audio (increase sample rate, control sample pitch, reduce CPU load and jitter)
- Support for creating and editing melodies and patterns
- Optimized graphic libraries – more fluent animations with reduced cpu load
- Selecting different instrument sounds
- Tuning parameters like velocity, bend, swing, ADSR and filters
- Various bug-fixing and test automation
- Stacked octave mode (several units form a single synth with more octaves)
- Wi-Fi connection to a generative AI server (done manually for the demo)
- Maybe a version with built in speaker?
- Add support for the AMY synth library?
Prices including freight (2025-01-05)
- 43 esp32s2 mini (129 | 3) (Amazon)
- 29 magnetic encoders (116 | 4) (Amazon)
- 26 ball bearings (105 | 10) (Amazon)
- 25 switches (100 | 5) (Amazon)
- 20 LEDs (78 | 100) (Amazon)
- 43 LCD (128 | 3) (Amazon)
- 22 battery charger (89 | 6) (Amazon)
- 10 battery (17 | 4) (Ali express 803040 1000mah)
- 25 DAC (100 | 4) (Ali express PCM5102A)
- 10 misc. diodes, power switch and filament
- 10 PCB (40 | 5) (JLCPCB)
- total: 263 SEK => $24
Download project files
All files required for this project are available at this GitHub repository:
https://github.com/vonkonow/LeetAI
License
This project is open source under MIT License
(Attribution is optional, but appreciated /Johan von Konow ;)