Actually If this was running Pure Data that seems like a match made in heaven. The organelle synth does something similar and it’s super snappy how you can program patches anywhere Pure Data will run, then copy them over to the synth and run them. You have to tweak the patches of course, to take advantage of the hardware, but https://www.automatonism.com/the-software has some GPL tools for that, and I bet you could avoid completely re-inventing the wheel by using those.