One thing to note: There's no need to use -windowgui in Windows 8. The menu and cutscenes display just fine! Windows 8 doesn't have color issues with paletted games.
Sorry to say but there seems to be no way to get the D3D mode working. Maybe someone will have a solution someday, but until then Software mode isn't too bad (again, since Windows 8 doesn't have color issues like 7)
You can limit the framerate using a ddraw.dll from http://bitpatch.com/ddwrapper.html
(This will also work for the D3D mode if that ever gets fixed later, I tested it in Win7)
Turn on FakeVsync and set FakeVsyncInterval to 34 to limit the game's framerate to 30. For some reason Jedi Knight will keep the game's framerate at 30 if you run any higher, but then the weapons will start to animate too fast. So 30 is the ideal framerate.
You need 'Backbuffer in System Memory' off in-game for it to work.
If it runs too slow (around 20fps) then you can fix it by editing the registry.
1) Run regedit.exe
2) Go to HKEY_LOCAL_MACHINE/SOFTWARE/Microsoft/Windows NT/CurrentVersion/AppCompatFlags/Layers
3) You should see an entry for JK.EXE. Double click it to edit it.
4) Replace the entry currently in it with
$ DWM8And16BitMitigation 8And16BitTimedPriSync 8And16BitAggregateBlts Layer_ForceDirectDrawEmulation
Layer_ForceDirectDrawEmulation: Uses software emulation instead of hardware emulation (Windows 8's default) which of course doesn't work well with the software mode (or at all...). Additionally this also prevents the Charms menu from showing when in-game.
8And16BitTimedPriSync: Menu items will vanish and cause strange artifacts in the menu without this when DirectDraw emulation is on.
8And16BitAggregateBlts: Game will flicker black without this when DirectDraw emulation is on.
DWM8And16BitMitigation: Windows will add this permanently if you run JK with a compatibility mode. This is what prevents the "need 256 colors" error.
Note that this will disable the option to use Hardware Acceleration, so if a fix for it DOES appear in the future, you'll have to open this key again and remove the "Layer_ForceDirectDrawEmulation" flag.