Some thoughts:
Keep in mind that supercaps can only use a tiny portion of their avalable energy capacity, because it is distributed across the entire 0v to max volt range of the cap, while a battery can use it's entire available energy capacity because it is distributed across *just* the working portion of the voltage range of the battery.
So...adding the same volume/mass of battery to the traction pack will get you more in both regen capability and traction capability than the caps will, if you are not using cells incapable of the necessary charge rates, especially if you are using the cells in a way that never charges them to full with the normal charging process, leaving some "room" for the regen charge.
For instance, if you use a "100v" system, the typical voltage range a motor controller requires is about 88v to 118v.
A "100V" battery's usable range is within that, so "all" of the available energy the have is available to the controller and motor. They start with energy at the 88v level, when "empty". (really they have some energy below this, but it is very little vs the rest of the capacity, and is less stressful on them to not use it)
The supercaps (presuming 130V+ capable so they won't be damaged by voltage spikes) usable range is 0v to the system max voltage (118v in this case). They still have to have a lot of energy placed into them to get them up to the system range, but the system cannot use any of that energy, below the 88v level. It can only access the energy between 88v and 118v, in this example.
Motor:
Unless you need engine enhancement / braking near the full RPM, you don't need a motor that can do that RPM, if it has a clutch between the motor and engine drivetrain.
Or you can use a gear reduction between the motor and the drivetrain so that it reduced the engine RPM fed to the motor in braking and increases the motor RPM fed to the engine during traction. The gear reduction can be as simple as the belt or chain that transfers the power to the engine drivetrain, by not having equal sprocket sizes, using smaller on the engine side and larger on the motor side, by whatever ratio you need the RPM to be changed.
For brushless motors, a high-RPM inrunner motor must be designed with IPM (not SPM) or else must have the magnets secured very well (like some of the FX series of motors that were wound with kevlar around them).
An outrunner motor's magnets will be held against the casing by the forces involved, so don't need special mountings/etc. But the outrunner's casing spins, so has to be mounted by the non-spinning portion of the core that sticks out for this purpose, and everything near the motor must be secured so it cannot touch the motor in operation.
Brushed motors are not usually suitable for high RPMs unless they were specifically designed for that, as the commutator can actually explode from the forces (sometimes called grenading).
You'll probalby want a brushless motor, and run it with an FOC controller, because those will give you variable regen control, whereas many other controllers only have on/off control of a preset level, which is very hard on all of the mechanincal components between the motor and the wheel if you need much regen braking force.
BTW, there are existing systems (Buick?) that use a motor in the alternator belt system to do what you want to do. If you can get a complete system from the scrapyard it might make your project simpler.
The big catch is space--the Mazdas I've seen under the hood (RX7) have very little space in there, and don't have anywhere you could put any of this stuff. If yours is like that, you'd have to put this stuff into the "passenger' area and run it down thru the floorboards to the driveshaft instead of the engine. That would also be easier on the engine/transmission, as it wouldn't have to handle the braking forces, or the drive forces if the transmission clutch doesn't allow backfeeding of wheel forces up into the engine.
