Dev Teams need to QA hardware/software configs more thoroughly

[Tryptonite]

This particular rant is not specifically targeted towards the Relic Development team, but all game Development teams in general. But I will be using DoW as an example for my argument.

I wish you guys would invest a little more time and effort in testing the builds more thoroughly on a larger variety of hardware/software setups. I understand that your QA department is probably doing this to some extent, but really - how large is the variety of PC setups among the QA testers? Is there a group that is specifically dedicated to QA'ing as many hardware setups as possible? Bottom line is that are you doing enough to be pro-active as opposed to reactive with these kinds of problems?

Granted that it is damn near impossible to emulate every possible setup and I wouldn't expect any QA team to, but if you're producing a PC game and that's your only platform, then there needs to be a special focus on really putting that game through the ringer. This whole ATI, AMD or soundcard stability issue is choking a lot of people up. I think to some extent that this issue could have been foreseen and prevented if there was a stronger focus on compatibility testing.

I appreciate and applaud the members of the Dev/QA team who are hitting this forum hard and trying to help people out. I'm glad to see you guys are following up and trying to get to the root of the problem. A lot of us here really do receive peace of mind knowing that we are being heard and action is being taken.

A lot of us are just waiting patiently for a patch. I hope there are at least a few members of the Dev team in this weekend to get started on the troubleshooting. Thanks for developing a great game. If I sounded angry or unfair then I apologize. I'm just getting frustrated at not being able to enjoy a great game this weekend.

peace
Tryptonite

btw: Just for the record I also work for a large developer and I am often disgusted at how poor the compatability testing is on certain PC titles.

[ÜberJumper]

"...if you're producing a PC game and that's your only platform..."

Can you spot what's wrong with what you just said?

[Bonnet]

The QA team at relic do not enjoy the fund nor time to test all of the avaible configurations. And so far it looks like alot of hte problems lie with seucrom which is not relics fault. They ran an extensive beta, and it worked on a ton of computers. I guarntee there was several AMD and Audgiy sound cars setups in it, so I don't think you can blame them.

[Nerevarine]

I don't know what happened during beta, but it crashes on everyone I know with nforce/ati combos. Doesn't seem like a weird combo to me, at least not one to be "missed" during QA..

[ÜberJumper]

zbobet:

Tryptonite is correctly refering to the publisher, who is supposed to do the most extensive QA work.

[Tryptonite]

"...if you're producing a PC game and that's your only platform..."

Can you spot what's wrong with what you just said?

I am referring to the fact that this game was developed for PC and not for other platforms such as PS2, XBox NGC etc. If I am mistaken and it is being ported for other platforms, then my apologies.

By common sense, QA'ing a product on multiple platforms is more difficult than QA'ing a product on a single platform. My point with that statement was that if you only had to QA the product on PC then your job should be more thorough and concentrated. Testing on multiple platforms can be a logistical nightmare and plays a much heavier burden on your resources and your QA team.

If I still missed your point Uber - please point it out to me. :Slap:

peace
Tryptonite

[ÜberJumper]

Tryptonite:

What I was getting at is that the PC is not anywhere near a single platform.

A small example...

Win98, Windows Me, Windows 2000, Windows XP, Windows 2003.
ATI 9000,9200,9700,9800,X800 Nvidia GeForce/2/2mx/3/4/4mx/5xxx/6xxx
Soundblaster Sound Cards, AC97 compatible cards, fortisimo cards
Pentium IV CPU, AMD CPU.
Intel Motherboard Chipset, Via Motherboard Chipset, Nvidia Motherboard Chipset, SiS Motherboard Chipset.

Now combine those in every possible way and tell me that the PC is a single homogeneous platform. Wait, before you do that, consider the obscure bits of software a user might install, or the bajillion possible video card drivers for a start, how about viruses, or other corrupt system files?

So we can see right away that PC QA could be BRUTAL to configure. This is the main reason why a developer would hate to develop games for the PC, simply because there's so much that can go wrong.

What QA labs logically would do is test the most common hardware configurations out there, and gradually, as time (and especially money) permits, test down on the less and less common hardware configuations. Sadly, that testing usually ends up well before every possible combination would result in being thoroughly tested.

Hardware Compatibility Labs therefore have to pick their battles. If say 5% of PC users are using a specific hardware combination, while 50% are using another combination, where would you devote your time?

I agree compatibility testing should be more thorough, but realistically, I know it can't be. Until there's more solid standardization between hardware, we'll always see incompatibilities.

[Tryptonite]

Tryptonite:

What I was getting at is that the PC is not anywhere near a single platform.

A small example...

Win98, Windows Me, Windows 2000, Windows XP, Windows 2003.
ATI 9000,9200,9700,9800,X800 Nvidia GeForce/2/2mx/3/4/4mx/5xxx/6xxx
Soundblaster Sound Cards, AC97 compatible cards, fortisimo cards
Pentium IV CPU, AMD CPU.
Intel Motherboard Chipset, Via Motherboard Chipset, Nvidia Motherboard Chipset, SiS Motherboard Chipset.

Now combine those in every possible way and tell me that the PC is a single homogeneous platform. Wait, before you do that, consider the obscure bits of software a user might install, or the bajillion possible video card drivers for a start, how about viruses, or other corrupt system files?

So we can see right away that PC QA could be BRUTAL to configure. This is the main reason why a developer would hate to develop games for the PC, simply because there's so much that can go wrong.

What QA labs logically would do is test the most common hardware configurations out there, and gradually, as time (and especially money) permits, test down on the less and less common hardware configuations. Sadly, that testing usually ends up well before every possible combination would result in being thoroughly tested.

Hardware Compatibility Labs therefore have to pick their battles. If say 5% of PC users are using a specific hardware combination, while 50% are using another combination, where would you devote your time?

I agree compatibility testing should be more thorough, but realistically, I know it can't be. Until there's more solid standardization between hardware, we'll always see incompatibilities.

I stand corrected regarding PC as a single Platform. I sometimes forget there is a myriad of components involved with the PC . And I do understand the difficulties involved with the whole PC QA picture you have painted.

It was more frustration than logical thinking that went into my post.

Thanks for clearing things up.

peace
Tryptonite

[ÜberJumper]

Your welcome, thanks for taking the time to follow up.

You work for a local developer? EA by chance?

[Tryptonite]

Your welcome, thanks for taking the time to follow up.

You work for a local developer? EA by chance?

Yeah. The studio in Burnaby focuses primarily on console platforms and handhelds and that's my specialty so I only have a basic understanding to the overall QA process for PC titles.

That's why I should have known better than to make broad sweeping comments without fully understanding the situation. I'm usually more reserved in my day to day communication. Funny what forums do to you. Heh heh =P

Keep up the good work Uber

Peace
Tryptonite

[Anakaris]

I personally blame the hardware industry in general. All competition and non-standardization make frustrations for everyone.

It would be like buying bolts for the Space shuttle. There are hundreds of different manufactures of bolts worldwide. Whereas that could lead to the statement "A bolt is a bolt right?". But in reality there are many different materials, machining processes, and tolerences that go into making a bolt. Not paying attention to those could make for big problems with such a high precision machine like the Space shuttle.

In a way this is happening for the computer industry right now.

[FatalTheRabbit]

I cant help but wonder what would happen if a more well known company like blizzard released a game as bugged as this one.

probably quite a sh*tstorm.

Ive also been thinking that its relics new publisher and not relic itself that is at the root of it all.

[Tryptonite]

I personally blame the hardware industry in general. All competition and non-standardization make frustrations for everyone.

It would be like buying bolts for the Space shuttle. There are hundreds of different manufactures of bolts worldwide. Whereas that could lead to the statement "A bolt is a bolt right?". But in reality there are many different materials, machining processes, and tolerences that go into making a bolt. Not paying attention to those could make for big problems with such a high precision machine like the Space shuttle.

In a way this is happening for the computer industry right now.


Using the bolts example is an excellent analogy Anakaris. I think consumers are starting to expect to see more uniform standards for PC hardware. We really want to see games and applications get to the level where you can pop it in like a PS2 or XBox game and everything works smoothly and is stable without any worry of config problems.

With the current situation that you and Urban have identified, it's going to be a long road to get there.

peace
Tryptonite

[ÜberJumper]

Trypt:

Mokona (Darwin) works at EA Burnaby.

Fatal:

THQ's only really started ramping up for High Quality PC Titles in the last year or so, they therefore likely don't have a massive QA / Compatibility labs department to do more compatibility testing. Hopefully, they can find and fix some of these hardware problems.

[Gigabomber]

It is very funny that you mentioned what would happen if blizzard released a game with a horrendous amount of bugs in it because i was thinking, since i got this game, that i had not seen a game as buggy since diablo 2 was released early.
I have found an interesting phenomenon across the board with car manufacturers, video game, just about anything you can think of; it comes down to a simple philosophy: why delay the release of a new product that you can get people to pay for and get the biggest, across the board feedback possible instead of delaying a game debugging it until the hype dies down? This would all work fine and dandy if the companies repair the products with speed and accuracy.
In the case of blizzard diablo 2 had a plethora of simple code errors, one of the funniest being a simple minus 1 in the code so that the maximum damage for weapons and defense for armors was not possible to obtain or perhaps the fact that 4 types of weapons would not drop at all.
As much as I hate sloppiness, the nature of capital is at heart here and with a demo or beta many people dont give feedback anyway its just a gearup for the maingame, its much worse when you get a new model car and are the "tester", trust me, the problem is that most of these games are based on new or revamped engines. Id software is about the only company I can think of that the games ran pretty solid right off, but i am sure that people cropped up too often with issues.
From the manufacturers viewpoint, consider this, its a 50 dollar video game, if you buy games new your likely to buy the next one they spit out thats new, i generally dont buy new because of these bug issues, i took a chance and was rewarded with no patch in a playable but unenjoyable scroll-lag-like bug.
Oh that reminds me, remember how long it was before they preloaded the info for the duriel battle in diablo 2? that alone was probably the lamest video game situation in history
P.S.sorry if you dont know any of the diablo 2 stuff but it serves as a prime example of what i am talking about with marketing and merchandise

[tempo36]

While I understand how varied computer combinations can be, it seems like many of the problems are occuring with prime hardware components...like 6800's and 9800's. With new AMD and new P4 chips....updated drivers. We're not talking about folks running with 3 year old drivers and old voodoo video cards. We're talking about the cards, chips and drivers that are practically named on most min/recommended spec sheets these days. That's what bothers me...these are problems occurring on brand new up-to-spec systems.

[ÜberJumper]

Yet there are clearly brand new up to date systems that DO run the game with no problems...

Funny that eh?

[tempo36]

Indeed. :)

[ÜberJumper]

Prime example here of something outside of Relic's control...

http://forums.relicnews.com/showthread.php?t=39354

There's also someone I've been dealing with on MSN who had a problem with his powersupply not providing enough power to dual channel memory under high load.

[TheDeadlyShoe]

Re: Powersupply

I think that's actually pretty common. The generic power supplies that come with the majority of computers are really pretty shitty in most respects. I've had several break down. Typical symptom when I noticed was hardware spontaneously refusing to work, general instability, constant crashes under any kind of load...

If you disconnect some peripherals and everything is hunky dory that's usually a good way to tell that you probably need to swap out power supplies.

I use much higher quality power supplies now, personally. They can cost as much as a generic case (WITH power supply), but it's worth it. (They're quieter and cooler, to boot.)

[Hellshot_69]

ummmm.... I have an ultra 600 Watt power supply. no power problems there ;) and yeah. Crash city.

[TheDeadlyShoe]

600 watts?

That, um, sounds unstable to me.

An' Wattage != Quality.

[Hellshot_69]

yes. Tested and true 600 watts. Busted out my handy ol fluke multimeter to verify. Some uber drawable power supply. Very hefty price tag as well. Check it out at tigerdirect.ca.

[ÜberJumper]

Again, that's just one example what the problem could be. PC Games development comes with no hardware standard, and bajillions of possible hardware/driver/software/user error combos.

PC Game development will go the way of the dodo if they can't get some more standardization happening. Too much work.

[Hellshot_69]

well, now that my other thread is locked.

Patch waiting it is.

[ÜberJumper]

There's no garuntee your issues will be fixed in a patch if the cause has not been identified.