General >> General JK and MotS Discussion >> Something to share (Shadows of the Empire)     (Moderator: Admins) Previous Topic | Next Topic
Page 1 of 1
 Author 
 Post 
Post 29-May-2011 22:34        

Please read on the second reply which is newer!

Old text:
So I just thought to come over here and do a visit do the good old jk hub
Sometime (either much more now) I wanted to write something about that topic but I had not the time for all. As I stumbled over an interesting project I did approx. one year ago, I think let me write about it.

I did something like a Map-Viewer for the Shadows of the Empire game by lucasarts (1997) and now I try to get this thing to work again. I must get some code fixed, because I do not have the old environment I did it on once. I need to get the Game first somewhere and then I could do some sceenshots. On the other hand I do not have the real time to improve it or to do anything for this, so yes I will release the source of the app too! It is written in C# and was compiled with VS2008. Like I said there needs to be some fixes done to get it working again, at the moment it does not compile. Currently I am converting it for the VS2010 C# as I dont have the previous version and the new one is pretty comfortable.
As I remember you can load any map from the game but I did not reverse engineered that much
then thus you have to manually enter the data offset of the "meshes" and the object count (or vertices count).
This is what I am going to do when I got the app running, to get all of that start offsets from the files and the counts too. So you don't need to do that, because that a lot of fiddling around with a hexeditor. (But if you want to improve that whole thing I would be glad!) And it only can load the map as a whole set, it can't differ of what is a "sector" or "object" unfortunately as I did not got so far. There is only that "feature", no textures or so. Some levels are just a "bunch" of objects stacked on each other, as I did not get behind the positioning or even more, if there is somewhat like that, like a script.

About the format well I am unsure. I know from the ingame debug mode that it handles with somewhat it calls "sectors". If you change the location of the player the number is changed then. The only thing I am aware of is that I got the vertices of the whole map which are sequentially stored. There needs to be done a lot of things I know.

The data itself is stored in binary format inside the games dataXXX.dll files located in the same directory where the game exe file is on. Each of these .dll stores a SotE mission/cut scene etc.

I will write here when I got anything on stage and release the full source and the information about the format.

... Removed I am going to release the source when I have a stable version of the app.


Greetings & Peace frequent

Last edited by frequent on 28-Sep-2011 14:25; edited 1 time in total
Post 28-Sep-2011 02:19        

Sorry I had much trouble in private life but now I hooked on that project again!!
I am currently rewriting the whole code for the tool. It acts now as a fully SOTE Map viewer with proper format reading. No more data "predictors" when loading the level, the polygon format has now been fully reverse engineered. Mr Green

Yep I have allot more time now so I move on to r.e. something more than just the "plain" map format. I do not promise to develop a whole tool to make own missions but it could happen that this is getting real. Just a matter of time.

Screenshots:
2nd level: Echo Base Mission

Thumbz:


Big:
http://imm.io/9FMx
http://imm.io/9FMT
http://imm.io/9FN2

If someone, maybe with some x86 assembly knowledge for r.e, wants to participate / help me with reversing the engine I would be afraid & happy about

Currently I am using ollydbg and ida dissassembler with some other tools when working on this project.

Some specs about the Shadows of the Empire (PC version) engine (+what I know so far through reversing):
Ok, the game was released back in 97 for win95 with need of a 3d accelerator and entirely designed & released for the N64 in 1996. A patch, the only one, (v1.1) was also released 97 and fixed up some small things with the graphic engine.
Btw I use the original non-patched version 1.0 when reversing.

The N64 version is obviously compressed as the rom has "only" 12 MB (+streamed music!) in comparison to the pc version which has altogether 39MB (without cd-music/menu bmps and .san movie sequences)!!

The music on the pc version is cd audio / consisting of 13 Tracks. They are the same (only extended / not looped) as on the n64 except the menu soundtrack which is not on the pc version.
The n64-menu music on the other hand is actually a part of john williams "Star Wars Theme/The Imperial Probe (Extended Version)".

Now to the engine (PC) itself:
The games main executable is the 3.5 Mb big shadows.exe in the games "/sdata" directory. This is the .exe I use to reverse.
The Levels are stored in separate dll's. Each dll has two functions which returns a pointer to the levels actual data. The data itself consists out of lists with pointers to other pointers or data. It's like a small filesystem. Each pointer-list has a proper format with additional informations. Only the 3D-Mesh format is fully reversed now. Anything else needs to be examined deeper.
The 3D Data from the current stand consists all of a Bounding-Box (8 x y z vertices) and the meshes itself. All meshes of a level are parted up into smaller units = 1 Mesh. I guess the Bounding-Boxes are the "sectors", but that is only a guess for now. We will see more later

Now some thoughts about the functional part of the engine.
Uses sectors like in Dark Forces and Sith-Engine (JK/M.o.t.S).
Stores meshes + bounding box of the map/objects parts as collection of vertices + index list.
Uses Textures from Dark Forces and stores them in the shadows.exe.
Some interacting is hard coded inside the engine itself (Bike Chase / Mos Eisley)
The Ordmantel "Train" and Bike Mission uses plain objects for the world itself, so no static map is being used as other "on-foot" missions. Anything there is being setup dynamically depending on the players current position.
All other "on-foot" missions uses static and pre-build maps.


Btw: I bought an older GPU (GeForce 8400GS) to test SOTE and use the driver version "266.58_desktop_winxp_32bit_english_whql". It runs without any issues. Also JK works fine with that card/driver too.

Peace

Last edited by frequent on 28-Sep-2011 14:50; edited 2 times in total
Post 28-Sep-2011 08:42        

What's the difference between a "map viewer" & a "renderer"? Would it be possible to extract/convert geometry from "Shadows of the Empire" to another game (e.g., "Jedi Knight: Dark Forces II")? I only ask because a SotE mod for JK &/or MotS would be amazing.

Also, which engine did they used for SotE & how does it stack up against JK/MotS's Sith engine?
Post 28-Sep-2011 14:20        

Hi

It is the same as it renders the vertices from the "level" or what supposes to be a "map" like structure at the end.
It also loads the vertices of "objects" like doors, player and enemies etc, but currently they are not positioned on their right place as in the game. They just stuck in the middle. I need to find the piece of data which store the position of a mesh/mesh-collection (= object)

I could extract the data to a plain text file but it will not be sorted in any way. Just a bunch of vertex information then, currently also without textures. Therefore I also need to reverse the exact information about sectors and what defines a sector and thus what is inside this sector then. Then you could sparse the whole map thing into smaller regions and thus make it more friendly for JK/MOTS (sith) But I noticed also that some "rooms" in SOTE are not exactly aligned to others like it must be in JK (adjoining of sectors/regions). In terms of the sith-engine I know when they are not right and exactly aligned they will be ignored by the engine and so produce a "HOM/blank backgound" then. But it is definitely possible to create a 3do out of it.. I have to look at the 3do format first before I can verify that..

Currently I only have reversed those Bounding-Boxes which I could describe as "sector"...

SOTE defines actually "sectors" as the nice ingame debug display thingy tells about it when enabled and you are on "on-foot" missions. For more classification I also read about sectors here: http://www.gamesetwatch.com/2009/04/postmortem_star_wars_shadows_o.php
at the section "Technology":
It says
Quote:
...The way it works is quite elegant, thanks to Eric and Mark. The world is subdivided into “sectors”—that is, polygonal regions defined by...


Those regions they mention are interesting and I will concentrate next time on this.

btw I like this interview/story it is very informative and helpful when reversing the game. Good to know this and that. There was also such a story about the sith engine when I remember right where they talked about the cog/script technology they invented for jk.

In terms of the SOTE engine itself, hmm I think (currently) it must be a mix of technology they had back then. I know from the story that they used tools they used for Dark Forces when creating the "preview" of a "on-foot" map. I also see when looking on the lifts/elevators ingame, they all have this dark-forces "moving floor" thingy attached to it. So when the elevator moves top and you are just below, you can call the elevator back by "activating" just the surface of the elevators "wall". If you have played or play dark forces you know all those elevators just look like moving big pices of rams coming out of the floor instead of a small platform like they look in jk. The only difference is, that all elevators are pre-desinged in SOTE (I hope that is the right definition) and do not produce this "ram" dynamically like in df.
Furthermore some textures are from df too, like they on echo base/hoth.

If you read information about SOTE in the web you will often encounter the occurrence of the word "multimedia project", awwww what a description eheh , so I think this has something so say.
From my current point of view about scripting I can only verify that some scripting/interacting has been hard coded inside the engine itself. This for example is the mission where you on a bike in mos eisley, the complete chase scene is hard coded and also the biggest "function" in terms of the size in bytes (that tells me IDA-Pro)


Lets summarize that: SOTE has a touch from dark forces +newer technology. It uses "sectors" like df and sith-engine does and occludes hidden rooms/parts.

Some missions when you fly/bike are completely different treated by the engine as the "on-foot" levels.

The flying missions may not that depend on "sectors" like other missions do.

The train "ordmantel" (also on-foot) mission and the bike "mos eisley" mission build up their environment world dynamically and must be deeper investigated. They both use also sectors!

It is not using portal rendering like the sith-engine, maybe the occlusion uses some part of that technique.
The Textures are stored inside the shadows.exe and some of them are adapted from df.

All interacting like pushing a button, elevator calls etc could have a dynamically setup (so not hard coded).

I keep on reversing, maybe some facts I state now are disproved then.

I will make a sample video of the tool and put that on youtube now
Edit: http://www.youtube.com/watch?v=qqdsN4ZQYHY
And I create a little website "blog" where I post all information I have / reverse in a more sorted way!

Last edited by frequent on 29-Sep-2011 18:18; edited 1 time in total
Post 29-Sep-2011 07:34        

Yes, please do. I'd enjoy following your progress with this.
Post 29-Sep-2011 16:11        

Hi,
Just some quick Info:

Get ready for my youtube channel relating to this project -->
http://www.youtube.com/user/revsoteprj
(I try to post videos regularly)

And I am currently working on the website + blog which is going to contain all info about the project and sote, stay tuned

Thanks
Post 02-Oct-2011 15:28        

Only a short message from me about my current status with the reversing.
Yes it needs a bit more time that I thought... ok anyway

So far I'm working on the entire "file format" from the beginning on. With "file format" I relate to the entire data stuff used inside the dlls.

There is a lot of byte swapping done from big to little-endian so that Intel computers can read the values the right way afterwards. I think they just used the data they developed for the n64 with the devkits (mips cpus) and tools. All that mips hardware use all big-endian aligned hardware and data handling.

+Update about Sectors:
Now the game really uses a kind of a sector, but these sectors seems to play more a role with the scene handling. For example: in the train stage, when the player passes to another sector, the train stops and he must jump to the next one and so on.
I made an experiment and set all sectors to a single number, for example 0, and just the handling got wrong but nothing with the 3D data itself, so anything kept looking at it is.
The entire sense or more the right use of sectors will be posted someday when got further.

New things found but in experimental stage and not-that-proven:
+Found level's background selector and color as rgb values.
+Thus every sector can hold a own color / background type / fog-distance and a kind of far-plane distance which will take in effect when the player passes into that sector. Looks cool and weird the same time when changed I think there is more you can change actually ...
++background types: None (just smear the pixels around/HOM Mr Green ), Color, or Color + twinkling Stars!!

That's all for now I keep on posting with status updates regularly

frequent
Post 11-Oct-2011 04:33        

Very interesting stuff. I'll be following your work also. I'd love to see what could be put into Jk even if it required more work like redoing all the textures.
Post 12-Oct-2011 20:57        

Hi,
Small update: I found the texture handler and its format. It uses indices values and sort of a palette.


Got a little bit busy but I keep updates rolling
Post 13-Oct-2011 16:44        

If you didn't know, the reason you see all the Dark Forces material is because when they made the game, the level designers actually originally made the maps in Dark Forces. They didn't have an N64 to test with yet, and code was quite early, so by doing that, the level designers could start earlier.

I wouldn't be surprised to find vast amounts of other Dark Forces-like things in the engine and code.
Post 27-Nov-2011 23:01        

Sorry for the long gab of time but I am going to cancel this project. This time now for good.
I lost all my projects due an incident with my hotmail (skydrive) acc, but hey I don't want to talk about much that, it just sucks.
Either it got hacked or the service deleted files for some for me unknown reason. I keep actually any files of my files on skydrive as I don't own a cd/dvd drive anymore nor I have an usb key to save those files. I just have one harddrive for system and one partition for music etc.
Yep I formatted and re installed windows 7 + the second partition to make the tool align anything right (New hdds needs that) knowing that all files are on skydrive so I could download them afterwards back. No I did not check for availability of my files just before I did not even thought on this possibility. Thinking like hey why should my files are gone there hey this is skydrive never had those problems with it before. The time between the last backup and re install was about one week or so, as I did nothing big on programming either on my other files or docs, so there was also no big difference to re-upload something before formatting.

This was like October and as I am hanging more outside now, I don't had the motivation nor the energy to keep this topic up to date shortly after. First the news are obviously no good news, they are sucking news. I think anyone who lost his whole data is kinda screwed up for the first time, I have also seen that on hub a few times, when ppl. lost their projects and so due any kind of data loss. Beginning from start, hmm, not this time.

When thinking about the spend time on reversing s.o.t.e. and what I could have been done else in that time... No not to understand it wrong, I had much fun on doing so, getting insight to the s.o.t.e engine and assembler language, building a tool on the other side...

Maybe this might a sign to stop activity on programming and diving more into social related things for my future, no I don't want to get too 'nerdy', as I am not up to this on my mind.
I am rather a healthy person doing much sports or walking longer ways in free time, but I have to watch my self when getting interest on computer related things like programming. I know that I am too excentric sometimes and spending a lot of time on development.

Sorry for those bad news, it is not easy to clear up when that happens but it should be more comfortable than saying nothing.

frequent
Post 09-Jun-2015 17:49        

SotE was a huge game of my childhood and I always thought it would deserve a proper remake and as I grew I learned this and that about modding and I played with the thought that perhaps I could try to make a mod for JKII or something. I wanted to make it as close to the original as possible, only with updated graphics. The problem is that I couldn't find a way to rip the models and maps from the game to have a starting point. I have tried pretty much everything I could with tons of google searches including 3dripperdx and vrml exporter for the 1964 emulator but I failed with everything, all I could rip was the textures. (And sadly I know nothing about reverse engineering exe and dll files either.)

That's when I found this thread and as I started reading I was sooo happy. Then as I got to the end and noticed that you have cancelled this and the whole thread is older than 4 years, it hit me right it the face.

Is there even a tiny bit of chance for you to return to this or at least give a few hints for someone who is willing to start over?

I thought that I'd write it down, because even if it's in vain, I have at least tried everything I could.
Help me, you are my only hope, like Obi-wan Kenobi to Leia:D
*** Post commands are unavailable for guests. ***