[TOOL] Space Marine Tools - UPDATED 18-09 - OPPCExtract v0.1.0 released!

[Santos]

Wow, sounds great Santos. While fairly unlikely, and probably a whole lot of work, how much of a chance is there that creating levels may be possible at some point? Since creating an actual level editor would be way too difficult, I guess it would have to be done as a max exporter or something though. At least if you make them XML readable then it would be possible to tweak things like weapon and enemy placement and quantity.

Well, it's more of a problem of understanding the data and how it all fits together, than witing the actual editor. Creating levels from scratch might turn out to be way to complex, but I expect doing some tweaks should be possible.

Also, more questions, have you got any idea how the ragdoll/cloth bits are done? I cant see any bones on the import, so I am wondering how the attachment point is determined? Also wondering, since the tabard thing has a thickness, if there is some sort of vertex painting or weights or some other info stored there? since I would have thought if the vertices were allowed to just fly around as they saw fit they would tend to intersect and the back would tend to pop through to the front etc?

I'm not yet sure how cloth simulation works. What I know so far is that there is a flag for each mesh, that specifies if mesh is static, skinned or cloth. For cloth meshes there is also a HKX file embedded inside DCM files, which probably defines some bones and other cloth simulation related parameters. Havok will definitely be required to create new cloth meshes, but at least Relic moved from the ancient version 4.5.1 to version 2010-2.0, which happens to be available for free via Try Havok program.

One last thing, dunno if its an issue or not but the scale seems unusually tiny, and at least in max 2009 makes editing a little annoying. The SM is about "2" generic units high, possibly 200 would be more in order?

That's the original scale. I could rescale each model upon import, but I'm not sure if this would be wise as there are some really big models in this game (and I mean really big like entire ruined city district! ).

[Carbon14]

At least with the levels as far as I can see all the lighting is dynamic, so with any luck the maps are a somewhat simple combination of vert/plane data and uv data with a layer of lights and prop models over the top. I'll cross my fingers that it's possible, as I am admittedly more of an environment artist than character artist! Adding actual new playable content to the game would be rather awesome

I am a little worried by your description of the havok data, it sounds like it might be a bit of trouble to get that working even if you do manage an exporter of some sort. I can't say I am terribly familiar with havok, was hoping more solid forms like hanging chains or hanging crosses would be possible if it was using a system of ragdolled rigged bones, but if its only cloth then I guess that would be a no-go! I'll have to check out the HKX file format, there seems to be a bit of info on the web!

As for the scale, maybe a dialogue could be added at some later date to multiply the scale? depending on the amount of work of course.. On the version of max I am using the camera handling and selecting seems to get a little messed up when trying to edit really tiny objects.

I know c# reasonably well, really need to learn how to create properly functioning applications so I could help with the stuff O.o

-more edits-!

Since you seem fairly familiar with the engine, speaking of the shadows it would improve the games appearance slightly if it were possible to turn the shadow map up higher than the highest setting. In games this is usually a simple value stored somewhere that causes the engine to decide what resolution and therefore how much vid memory is used by the shadow maps. I'd be interested to know if you have found where such info is stored

[Santos]

At least with the levels as far as I can see all the lighting is dynamic,

Yes, I remember this being mentioned in some interview with Relic employee. They were happy about it because it simplified level creation.

I am a little worried by your description of the havok data, it sounds like it might be a bit of trouble to get that working even if you do manage an exporter of some sort.

Havok comes with it's own exporter so it's not a problem at all.

I can't say I am terribly familiar with havok, was hoping more solid forms like hanging chains or hanging crosses would be possible if it was using a system of ragdolled rigged bones, but if its only cloth then I guess that would be a no-go!

No, no, it's not only cloth. There are other Havok files (not embedded in DCMs) which probably contain ragdolls and things like that. I am also able to import rigid bodies from PPK files at this point. I know the game is also using Havok Behaviour which is a fancy system for controlling object's motions with the use of state machines.

I'll have to check out the HKX file format, there seems to be a bit of info on the web!

HKX is not a strictly defined format, it's more of a container for various kinds of assets (meshes, animations, rigid bodies etc.). For example in DoW2 HKX files only hold animations and rigid bodies, but models have their own format. What Space Marine's HKX files contain, it remains to be discovered...

As for the scale, maybe a dialogue could be added at some later date to multiply the scale? depending on the amount of work of course.. On the version of max I am using the camera handling and selecting seems to get a little messed up when trying to edit really tiny objects.

You would have to use the exact same scale for exporting. Well, I will consider this but only after importer is fully operational.

Since you seem fairly familiar with the engine, speaking of the shadows it would improve the games appearance slightly if it were possible to turn the shadow map up higher than the highest setting. In games this is usually a simple value stored somewhere that causes the engine to decide what resolution and therefore how much vid memory is used by the shadow maps. I'd be interested to know if you have found where such info is stored

There are some files with WMEM extension, which seem to contain some memory related stuff. Can't really remember anything specific about shadows though.

[badgut]

santos i have some troubles with trying to get Space marine models into 3d studio max.

I did the script thingy and the readme perfectly told me where to instal and all that stuff but not for the formats or the smoppc I would like to ask you where to instal that.

[Santos]

Formats directory just contains some descriptions of file formats used in Space Marine. You don't need to do anything with it, it's for those modders who want to write tools or do some more research. smoppc is a command line program, it doesn't need to be installed. See here for more detailed explanation on how to use it.

[badgut]

Thanks I did the unpacking bit but somehow my 3ds max cant import them.
Do thee perhaps knw the answer on that?

Edit: never mind found out how Thanks Santos!!!

[Santos]

Perhaps...if you tell me what exactly happens when you try to import a model?

[badgut]

Santos the problem was I didnt used the Space marine importer xD.
Basically I want to make an space marine from all the armor parts but everytime when i try to import 1 thing the other thing that is there gets removed, If you could find that out for me then your my angel.

[Santos]

Uncheck Reset Scene. Problem solved.

[Shuma]

Baw Gawd, i leave for a couple of weeks and return to see glorious Space marine tools and already people playing around with models and textures. It's kind of like Christmas, awesome work.

[NastyNoodle]

Shuma how are ya! totally ignoring my thread too, the cheek ;3

And you totally never accepted my invite after I whupped you in dow2 :P

Check your friend invites

[Santos]

Shuma ya lazy git! Good to see you!

[Shuma]

@Noodle:

Nonsense madam, i just posted on it, i just had a lot of threads to catch up with. As for Steam, checked and accepted but i must let you know, i'm rarely online on steam so it's not that i ignore people, it's just that i'm never there.

@Santos:

Aye good to see you too, i must say i think it's amazing that we're already getting tools here, i shall be monitoring this thread.

[badgut]

Thx Santos you are my angel

[citadel535]

Santos, thank for sharing this. Running around in this game makes me wish for a Wh40k rpg but with some clever use of modding I can make a cheesy experience. Any chance you know how to extract the sound files? The chainswords sound great.

[Santos]

Haven't done any research on sound archives yet, but the format doesn't look overly complex. I will try to crack it over the weekend.

[Kresjah]

@Santos: Considering you said that you're reworking the normals import this might not be applicable anymore, but perhaps this information will still end up being useful. At the time I posted about the normals issue, I hadn't gotten anyone else to check it on their setups yet. Carbon14 stated that it worked fine on his computer, so that pretty much means local issue of sorts. After a while I got a sneaking suspicion and got someone else to test whom ended up having the same problem as I had. Whilst it seems very likely I realize that it doesn't necessarily mean that it is the source of the issue, but I asked him in particular because I knew we both had one thing in common with our setups; We both had CAT installed.

I do recall that the old tools for DoW1 had some issues when CAT was installed, albeit I seem to recall that this was due to variables having some naming collisions (might be wrong; fuzzy memory). Don't think the error message was the same either.

Again, don't know if this is useful at all. If it should be and you need some additional info, do tell and I'll try to supply it.

[Santos]

Thanks Kresjah, I will investigate this.

In the meantime, I have another tool for release.

Download Links

Download from Hotfile
Download from Rapidshare

This tools is called smbod and it's a simple command line application (at least for a time being). It can convert so called BOD files into XML and back to binary format. BOD files have different extensions which are listed below:

• bmat
• O3d
• object-manifest
• PSystem
• Layer
• region
• world
• ssdecal

Program can be run as follows:

smbod [-d] [-c extension_list] [-x] [-b] source_path

where:

-d - decompress BOD file (or files if directory provided as source_path)
-c - compress BOD file (or files if directory provided as source_path)
-x - convert from BOD to XML
-b - convert form XML to BOD

As an example, to convert one bmat file to XML you would do something like this:

> smbod -x material.bmat

This will create a file called material.bmat.xml in the same directory where original file is.

> smbod -x c:\packages

This will convert all BOD file in c:\packages directory into XML.

Note some of the BOD files are internally compressed (in addition to general OPPC compression). These included following types of files:

• O3d
• Layer
• region
• world

You can decompress them with -d flag and compress back with -c flag like this:

> smbod -d object.O3d
> smbod -c "O3d" object.O3d

Note that when using -c flag you need to provide explicit list of file extensions:

> smbod -c "O3d Layer region world" c:\packages

Command above will compress all files in c:\packages directory with extensions included in argument to -c flag.

The last feature is converting back from XML to BOD format, which can be done like this:

> smbod -b material.bmat.xml

Program will strip xml extension and save the converted data to original material.bmat file.

The only problem here is that I'm not able to generate exactly the same file as original one, byte to byte. Each BOD file contains a string table in which every string has a hash assigned. I belive these hashes are used as keys for faster lookup. The thing is, exact method of generating those is unknown, so I had to code my own but it does not produce exact same numbers. Now, my theory is that the game should still accept those as long as they are unique but this needs to be tested obviously.

So I'd like to ask someone to do a simple experiment. Convert some material file (bmat) to XML, do some simple changes (swap diffuse and normal textures for example), convert it back to BOD and than use Cope's method to see if it works in game.

[Don the Oiler]

The -b command does nothing for me. I tried it with the sm_torso.bmat from Leandros' armour, it converts to xml with -x correctly, but when i try -b sm_torso.bmat.xml, it stays xml. No error message, nothing.

EDIT: Same with his sm_backpack.bmat. I tried renaming, deleting and keeping the old .bmat when i used the command.

[Copernicus]

@Santos:
I had a go at the BOD (BinaryObjectData) format a few days ago and managed to crack it as well. I'm still not sure what the DWORD after the zipped-flag is used for, any ideas? I also gotta say that I found the BOD format too complicated for what it is. Another question: What do the BOD DataTypes 05 and 07 mean to you? 07 seems to be some kind of index, 05 appears to be some kind of container which adds a key to an unkeyed value.

Edit: So after converting a file using your tool I see that you used 05 to create the element nodes and 07 to get their IDs.

[Santos]

@Don the Oiler

Sorry, my mistake. Links are updated now. Please redownload fixed version.

@Copernicus

I had a go at the BOD (BinaryObjectData) format a few days ago and managed to crack it as well. I'm still not sure what the DWORD after the zipped-flag is used for, any ideas?

Possibly version or number of objects in the file. It's equal to 1 in all demo files, so there's no way to tell really.

I also gotta say that I found the BOD format too complicated for what it is.

Agreed. Almost drove me mad at one point...

What do the BOD DataTypes 05 and 07 mean to you? 07 seems to be some kind of index, 05 appears to be some kind of container which adds a key to an unkeyed value.

I'm not sure about 05. You can read it as hashed string, but 0F is also a hashed string, so there must be some difference between the two. 07 is quite interesting bacause it's globally indexed. The value after 07 is starting from 0 and increasing by one each time 07 element appears anywhere in the file. Not sure what to think of this tbh.

Edit: So after converting a file using your tool I see that you used 05 to create the element nodes and 07 to get their IDs.

Not quite. I understand 07 to be element node and 05 the name of this element. Tbh it's still doesn't make terribly lot of sense but works good enough. I'm actually using the XML structure from the couple of files I've found in the demo (data\config\graphicsdb.xml and *.wmem).

Here's the full format for you: (feel free to fill in the blanks if you can )

================
BOD File Format
================
Version: 0.1.1
Author: Santos
================


Types
-----


GUID		unique id (8 bytes)
HSTRING		hashed string (with GUID being a key into string table)


------
HEADER
------


CHAR[3]		signature (BOD)
UINT8		? (0xFD)
UINT8		? (2)
UINT8		compressed (0, 1)
UINT32		? (1)
IF (compressed == 1) THEN
	UINT32		uncompressed data size
	DATA[N}		zlib compressed data (N - to file's end)
ELSE IF (compressed == 0) THEN
	UINT32		num strings
	UINT32		max string length
	FOR (num strings) DO
		GUID		id
		STRING		string
	OBJECT			root object


------
OBJECT
------


UINT8 		type? (1)
HSTRING		class
UINT32		num properties
FOR (num properties) DO
	HSTRING		property name
	PROPERTY	property


--------
PROPERTY
--------


UINT8		type
CASE (type) OF
	0x02:	INT32		value
	0x03:	FLOAT		value
	0x04:	BOOL		value
	0x05:	HSTRING		value
	0x07:	UINT32		id
		PROPERTY	property
	0x09:	UINT32		count
		UINT8		flag (0, 1)
		IF flag == 1:
			UINT8		type? (5)
			HSTRING		name
		PROPERTY	property
	0x0B:	UINT32		count
		FOR (count) DO
			PROPERTY	value
	0x0D:	DATA[8]		value
	0x0F:	HSTRING		value
	0xFE:	// No value
	0xFF:	UINT32		value


--------------
Property Types
--------------


0x02 - int
0x03 - float
0x04 - bool
0x05 - ?
0x07 - object
0x09 - array
0x0B - vector
0x0D - ?
0x0F - hstring
0xFE - ?
0xFF - ?


-----
Notes
-----


- if compression is equal to 1, entire data block is compressed with zlib (best compression level)
- four out of eight BOD file types are compressed - Layer, O3d, region, world
- property type 0x02 is signed integer (can be negative)
- property types 0x05 and 0x0F are both hashed strings, difference between the two is unknown
- property type 0x07 is indexed globally
- property type 0x0D is of unknown value type
- property type 0xFE is unknown and has no value
- property type 0XFF is unsigned integer, purpose unknown

[Don the Oiler]

Thanks. Repack works now, but the game seems to ignore the edited bmat, just loads the default one.

[Copernicus]

@Santos:
Thanks for all the information.
I believe the fifth byte of the BOD is the file version. Your header seems to miss a field: Right after your last unknown is another DWORD which seems to vary greatly from zipped to unzipped files. For zipped files, it could be the size of the unpacked data. For unzipped files, it may be the total amount of entries in the object tree.
IIRC 07 appeared only in conjunction with 05. 05 is very different from 0F as 0F comes in different flavors: Either type-value (as in an array) or key-type-value. 05 on the other hand never has a key afaik.

[Santos]

@Don the Oiler

Hmm, that's not good. Without getting these files to load, we won't be able to do any modding at all. I wonder if there's anything in the logs about this file failing to load?

@Copernicus

I believe the fifth byte of the BOD is the file version.

Might be. But again it has the same value in every available file, so we can't be sure. Anyway I'm perfectly fine with some values being unknown, as long as they do not change across different files.

Your header seems to miss a field: Right after your last unknown is another DWORD which seems to vary greatly from zipped to unzipped files. For zipped files, it could be the size of the unpacked data. For unzipped files, it may be the total amount of entries in the object tree.

This is the uncompressed data size but it only appears if file is compressed. For uncompressed files it is the number of strings in the string table.

BTW. Cope can you confirm what Don the Oiler said, that BOD files generated with my tool won't load in game?

Edit: Updated the format.

[Don the Oiler]

There's a log folder, but it doesn't contain any files.
The game's preloading on steam now, looks like smoppc and unpack don't like the full version oppcs (wrong file signature).

[Santos]

Oh? That's interesting. Maybe they are encrypted? First four bytes should say OBPK, else you get wrong signature error.

[Don the Oiler]

All the full-game files start with :0 , no OBPK to be found. The demo files have OBPK though.

[Santos]

Most likely compressed.

[Copernicus]

@Santos:
No, the game does not load your BOD files. It is definitely looking at them (according to ProcMon), but it'll still use the ones from the archive. Deleting the archive won't help, the game won't use your files (and instead load nothing at all).

[Santos]

Ok, well that is really not good. It seems there might be something more to those damned string hashes after all...

[Carbon14]

It's seemingly common practice to encrypt preloaded steam files so you cant mess around with them, or at least that's what I believe Valve usually do with their own games.

Glad to see more progress! Any updates on an exporter? or is that something that's out of your expertise Santos?

[Imperial Honour]

Yes the Steam pre-load files are encrypted at the moment, and they will likely not become decrypted until the release of the game (as standard procedure goes). So testing the community made tools on a Steam pre-load is pointless at the moment due to the fact that Steam encrypts the pre-load to protect it from pre-release piracy. By the sounds of it we will have a 0-day patch at the release of the game anyway, so god only knows what new content, possibilities or limitations we could possibly get with it.

[Santos]

Any updates on an exporter? or is that something that's out of your expertise Santos?

Well, I've done this before, you know. The thing is, for exporter you need to understand what every byte in the file means, so it's not exactly trivial. I've made an attempt already on writing back some files (couple posts above), but can't get it to work in game at the moment.

[Carbon14]

Ah, awesome! Sorry to sound demanding, was just curious because I have already started on a couple of custom models (power axe replacement and shoulder pad) and was selfishly wanting to know if it was looking likely that I may at some point be able to get them in-game

It also seems you have delved further in to the level parts of the game, and does Psystem stand for what I think it does? Particle Systems? Would definitely be interested to know if that is editable, space marines in snow and rain would be epic

[Shuma]

I got 2 questions, one pretty random but i've wondered about it since i started modeling on DoW II, why are meshes always blue?

And the other, what's the polycount of the generic marine? Like this one:

http://forums.relicnews.com/showthre...post1045286062

[Santos]

@Carbon14

PSystem stands for Particle System indeed and it defines game's visual effects. The good news is, this is still the same system as the one used in DoW2, CoH and DoW (with some additions like screen space effects) and I even happen to have editor for it. It needs some extra coding to operate on new file format but should be usable for this game eventually. Right now, if you want to see what's inside, you can use the smbod tool to convert PSystem files into XML format.

@Schuma

I can't quite recall why I'm using this particular color. I think the DoW sample models we got with mod tools, were like that and I just liked the color. If that bothers you much, I can tell you which line to comment out, to get your models in all colors of rainbow.

[Santos]

I have some bad news concerning Havok. I did look at the files and it turns out that despite having access to the same Havok version as used in the game, we won't be able to do anything with it. The game is using two Havok components - Behaviour and Cloth - which are not included in the free package available via Try Havok program. Every Havok file in the game is referencing one of those and this means I can't get them to load or even dump to XML...

So, this basically means we won't be able to modify or create new animations and it won't be possible to create new cloth items (loincloths, banners etc.).

[Nightshade]

that sucks, i was looking forward to another modding from this.

What about the models, any updates on those, for the full game?

[Shuma]

Thanks for the pictures it's around what i expected, pretty cool, as for the color it doesn't really bother me, in fact i also mostly use shades of blue when i apply a random material to my models, i was just curious.

Too bad about the Havok thingy, guess i'll never get to make a cape for Spezz Muhreen, but then again i'm more of an RTS man.

[Santos]

What about the models, any updates on those, for the full game?

I have a basic exporter almost done.

Too bad about the Havok thingy, guess i'll never get to make a cape for Spezz Muhreen, but then again i'm more of an RTS man.

I had even more grand idea!


Oh well maybe next time...

[Santos]

Ok, so I can export models now but need someone to test it. Here's a bolt pistol model replacement with some small additions:

https://rapidshare.com/files/868814773/boltpistol.dcm

If it works post a screenshot please.

[Solar Macharius]

Worked fine for me.

Nice Chapter specific modification

Images:

http://imageshack.us/photo/my-images...091300003.jpg/

http://imageshack.us/photo/my-images...091300002.jpg/

[Santos]

Praise the Emperor!

It works!

For those who may not realize what happened here, I've added a ultramarine symbol to the bolt pistol side and exported it from MAX with my own little exporter. In short we can put new models in game now!

[Shuma]

I guess it's time to give horns to Chaos Marine. Oh wait... Can we use mods with Space Marine and keep the online active? Cause i just noticed in that screenshot online is locked.

[Santos]

It must be due to this preview mode we're using to replace files. There's currently no way to pack the archives back unfortunately.

[Tin]

Just to stop any unneeded worry/work, apparently there's some places in Europe that have got the game prior to official launch date (Friday for us UK'ers), with the Multiplayer being locked until Friday. Maybe thats whats happened in the screenshot rather than the model being the problem

[NastyNoodle]

im fairly sure thats the case guys, same thing happens to me with or without my mod, but santos!! pleeeaaaaaase send me the files to export and ill love you long time lol

[Santos]

Soon™

[LoweGear]

Just to stop any unneeded worry/work, apparently there's some places in Europe that have got the game prior to official launch date (Friday for us UK'ers), with the Multiplayer being locked until Friday. Maybe thats whats happened in the screenshot rather than the model being the problem

I can say for certain that this isn't the case. I tried using NastyNoodle's "usepreview" skins on the full game, and while they work, the Online selection gets locked out if you do so. Basically with the current "-usepreview" methods, skinning the game will lock out the multiplayer.

[Mirage Knight]

I guess it's time to give horns to Chaos Marine.

This. For MP if possible. PLEASE.