FWIW, spoke size is determined by the rim strength, rather than the motor used. So the size you have is probably better for the typical bicycle rim than anything larger.
Even if the rims will "fit" larger nipples, the tension required on the larger spokes may damage the rim, often deforming or cracking it around the nipple holes, which leads to nipple loosening and then improperly tensioned spokes that don't carry the load, placing their share on others instead, that then place higher loads on their part of the rim, in a potential cascade of failures leading to a broken wheel.
If a rim is built for higher tensions then it can be used with spokes requiring them, but if it doesn't have manufacturer specs indicating it it's safer not to do this.
If the bike/etc weight on the wheel, or torque from the drive source, is higher than the spokes or rim could handle, then you would need to pick a rim that can handle the required spoke tension for them, *and* the weight/etc, and then pick spokes appropriate to these.
Reference my heavy heavy-cargo SB Cruiser trike for performance of Sapim 13/14 butted spokes radially laced to hubmotors in 20" wheels. There's also a set of posts in there in the last year or two about an OEM wheel build that either didn't use good quality spokes, incorrect rim for them, or incorrectly tensioned, where half the spokes just "exploded" during a normal ride home after only days (IIRC) of having the wheel installed, which has never happened to me in years of using the same motor type on the same controller (or even bigger ones with higher power, and also much heavier loading).
Regarding the frame / dropout spacing: You can probably cold set the frame wider***, but as you spread them the dropouts are no longer parallel to each other. You may need to bend them relative to the frame to put them back parallel, to be able to correctly fit the axle spacing and retention hardware.
Alternately you can replace them with torque plates with clamping / pinching dropouts, whcih is what I would recommend for a motor like this if you're feeding it with a big enough controller and battery to actually use it's capabilities, most especially if you are using any form of regen or ebraking at all.
Typical torque arms are pretty crappy and do not correctly secure the axle against movement--the holes are usually so large you can easily slip them over the axle, isntead of having to gently tap them onto the axle and down the length of it till they flush fit with the frame/dropout plates. If the flats of the retaining hardware are not fully parallel to and in contact with the flats of the axle, it will move under torque, and damage the axle and the hardware, eventually resulting in spinout of the axle and cable damage, which if shorts occur in the cabling can destroy the controller and the motor hall sensors. The most secure way to prevent axle movement is a clamp that is fully parallel to the axle surface on both sides for as much of the axle length as possible, for the least stress per square millimeter of contact surface area in torque transfer.
If clamping dropouts are not possible for whatever reason, very secure torque arms like the most recent versions ebikes.ca has to offer (not clones of them, but the actual ones they sell directly, which will actually be manufactured correctly) are your next best bet.
*** spreading the dropouts is most safely and precisely done using a threaded rod with fender washers against the inboard sides of the dropouts and nuts inboard of those, so you can turn the nuts until it is spread a bit wider than you need as the steel will relax once the nuts are loosened back towards the center. Results are controllable and even somewhat reversable as you can then put the nuts and washers on the outboard sides to unspread them a bit if necessary. Just start by spreading them to the point you need them to be, loosen the nuts and see where they end up, then repeat by expanding a bit further and loosening until they stay at the distance you require.
Note that it does weaken the points at which the frame is bending, but with steel it's much less likely to cause you a problem over the life of the frame than aluminum.
