How to use Linux kernel driver bind/unbind interface for USB-HID devices?

First background. I am developing a driver for Logitech game-panel devices. It’s a keyboard with a screen on it. The driver is working nicely but by default the device is handled by HID. In order to prevent HID taking over the device before my driver, I can blacklist it in hid-core.c. This works but is not the best solution as I am working with several people and we all have to keep patching our HID module which is becoming a chore, especially as it often involves rebuilding initramfs and such.