614955 - Enable unit tests on WinXP machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

We should enable them now that we can almost can.

We deployed Mozilla Build to all XP slaves in bug 549458#c38.
In bug 563036 we deployed VC 2005 SP1 redist which enables us to run optimized unit tests.
The Debug CRT will be deployed in bug 562459.

We will have to verify that we don't hit bug 592806 which so it seems it was specific to staging slaves (after re-imaging the unit tests run).

Comment 1

[Armen [:armenzg]]

I will be now be pushing this to completion.
I expect this to be done in the next 2-4 weeks.

Comment 2

[Armen [:armenzg]]

All XP test machines already have the "vc2005sp1_redist" already installed (https://bugzilla.mozilla.org/show_bug.cgi?id=563036#c17).

I am running all test jobs through staging and the optimized jobs are succeeding.

We could enable opt XP unit tests next week if we want to.

Comment 3

[Phil Ringnalda (:philor)]

dbaron: would opt WinXP tests be enough to let us shut off running tests on debug Win2K3 build slaves?

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Yes.

Comment 6

[Phil Ringnalda (:philor)]

Marking as blocking the "and disable them on" half of bug 614956, then.

Comment 7

[Armen [:armenzg]]

Thanks philor for updating the bug.

WRT to comment 2. I left on Friday happy seeing two test runs go green but today I was dissapointed to see purple runs on staging; I thought the machine got wedged but to my surprise there might be something else going on.

Only mochitest is failing to some odd reason. All the other test suites run to completion.
I will try to run it manually when I get a chance and see what's going on.

FTR here is one of the mochitest logs:

python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --total-chunks=5 --this-chunk=4 --chunk-by-dir=4
 in dir C:\talos-slave\mozilla-central_xp_test-mochitests-4\build (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', 'mochitest/runtests.py', '--appname=firefox/firefox.exe', '--utility-path=bin', '--extra-profile-file=bin/plugins', '--certificate-path=certs', '--autorun', '--close-when-done', '--console-level=INFO', '--symbols-path=symbols', '--total-chunks=5', '--this-chunk=4', '--chunk-by-dir=4']
...
args: ['C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\bin\\xpcshell.exe', '-g', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\firefox', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = 'c:\\\\docume~1\\\\cltbld\\\\locals~1\\\\temp\\\\tmpg1lozh';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='127.0.0.1';", '-f', './server.js']
INFO | runtests.py | Server pid: 336
args: ['C:\\mozilla-build\\Python25\\python.exe', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\mochitest\\pywebsocket/standalone.py', '-p', '9988', '-w', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\mochitest', '-l', 'C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\mochitest\\websock.log', '--log-level=debug']
INFO | runtests.py | Websocket server pid: 3764
INFO | runtests.py | Running tests: start.

args: ['C:\\talos-slave\\mozilla-central_xp_test-mochitests-4\\build\\bin\\certutil.exe', '-N', '-d', 'c:\\docume~1\\cltbld\\locals~1\\temp\\tmpg1lozh', '-f', 'c:\\docume~1\\cltbld\\locals~1\\temp\\tmpg1lozh\\.crtdbpw']
Traceback (most recent call last):
  File "mochitest/runtests.py", line 759, in <module>
  File "mochitest/runtests.py", line 756, in main
  File "mochitest/runtests.py", line 615, in runTests
  File "C:\talos-slave\mozilla-central_xp_test-mochitests-4\build\mochitest\automation.py", line 818, in runApp
    certificateStatus = self.fillCertificateDB(profileDir, certPath, utilityPath, xrePath)
  File "C:\talos-slave\mozilla-central_xp_test-mochitests-4\build\mochitest\automation.py", line 485, in fillCertificateDB
    status = self.Process([certutil, "-N", "-d", profileDir, "-f", pwfilePath], env = env).wait()
  File "C:\talos-slave\mozilla-central_xp_test-mochitests-4\build\mochitest\automation.py", line 173, in __init__
    universal_newlines, startupinfo, creationflags)
  File "c:\mozilla-build\Python25\lib\subprocess.py", line 593, in __init__
    errread, errwrite)
  File "c:\mozilla-build\Python25\lib\subprocess.py", line 793, in _execute_child
    startupinfo)
WindowsError: [Error 22] This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem

command timed out: 1200 seconds without output

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]

Comment 8

[Armen [:armenzg]]

I installed the CRT manually and it worked.
staging-opsi still was reporting the old talos-r3-xp-002 (*sigh*) which got re-imaged back in September.
I will have to fix the entry for that slave, deploy through OPSI and try again.

Comment 9

[Armen [:armenzg]]

Attachment: twisted/internet/_dumbwin32proc.py - be able to kill processes on XP machines

The XP machines don't have the _dumbwin32proc.py patch as documented in:
* https://wiki.mozilla.org/ReferencePlatforms/Win32#Downgrade_twisted_and_apply_process_group_patch
* https://wiki.mozilla.org/ReferencePlatforms/Test/Win7_64-bit#Buildbot.2FTalos_toolchain

I have also compared locally on C:\mozilla-build\python25\Lib\site-packages\twisted for winXP, win7 and win7x64 slaves.
The last two have _dumbwin32proc.py patched unlike on WinXP slaves.

I have also verified that twisted\spread\banana.py is already patched in all 3 platforms (as seen in http://hg.mozilla.org/build/twisted/raw-rev/0b441f0f68b4).

Comment 10

[Armen [:armenzg]]

Comment on attachment 499063 [details] [diff] [review]
twisted/internet/_dumbwin32proc.py - be able to kill processes on XP machines

Changing the review to feedback as I still have to verify that the patch does fix the issue (it most likely will as it is the same thing that is used for all the other windows platforms).

Comment 11

[Armen [:armenzg]]

Comment on attachment 499063 [details] [diff] [review]
twisted/internet/_dumbwin32proc.py - be able to kill processes on XP machines

This patch is not good enough. There is more than that.

Unless you have any strong objection I will pass the patch review to coop when it is fixed.

Comment 12

[Armen [:armenzg]]

Attachment: [WIP] twisted/internet/_dumbwin32proc.py - be able to kill processes on XP machines (v1)

This is what I am currently running on talos-r3-xp-00[1-4].
It matches what win7 slaves are running (both are running twisted 10.1.0).

So far I am capable to "stop build" the jobs.

Comment 13

[Armen [:armenzg]]

This is how I am deploying it:
 wget -Osubproc.patch --no-check-certificate https://bug614955.bugzilla.mozilla.org/attachment.cgi?id=499149
 cd C:\mozilla-build\python25\Lib\site-packages
 C:\mozilla-build\msys\bin\patch -p1 < "C:\Documents and Settings\cltbld\subproc.patch"

STATUS UPDATE:
##############
* we can enable optimized unit tests for XP since the CRT was deployed long ago *but*:
   * we don't have the twisted patch on the XP slaves as we do for win7, win7x64 and w2k3 slaves

PLAN:
#####
* step1. Deploy twistd patch - If the attached patch is the right thing I would like to deploy it tomorrow if coop_buildduty agrees and the whole run on staging goes well over night.
* step2. Enable opt unit tests for XP - If no fall-outs happen during Wednesday then we can enable optimized unit tests for XP on Thursday's downtime

NOTE: If any of the above fails to stick this will have to wait until January as I am going on holidays this Friday.

Comment 14

[Armen [:armenzg]]

I am not having too much luck.

The test runs are being able to be stopped but the machine is not being able to reboot.

The machine gets into reboot mode but it gets stopped with a Window saying:
"cmd.exe - Application error; The application failed to initialize properly (0x0000142)"

Once you press "OK" the machine is capable to reboot.

From looking at the twistd.log the last command being executed is:
> self.process.signalProcess(self.KILL)
http://mxr.mozilla.org/build/source/buildbot/slave/buildslave/runprocess.py#728

I am also comparing the diff between what we run on the slaves and twisted's 10.1.0 file (http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.1.0/twisted/internet/_dumbwin32proc.py) and I can only see a difference:
https://bug543626.bugzilla.mozilla.org/attachment.cgi?id=451277

Which is the first patch that I attached which is the same that is used on win7 and win7x64 slaves.

There is something odd for me. Why is it that I needed a second patch? If twisted 10.1.0 is what is installed on the win7 as well as the WinXP machines why did I need the second patch to match the code on win7 and WinXP? Not sure If I have over looked something.

I guess I should go deep into the patch and try to understand why is there the need of a patch to begin with.

2010-12-21 14:43:44-0800 [Broker,client]  startCommand:shell [id 20802]
2010-12-21 14:43:44-0800 [Broker,client] ShellCommand._startCommand
2010-12-21 14:43:44-0800 [Broker,client]  python tools/buildfarm/maintenance/count_and_reboot.py -f ../reboot_count.txt -n 1 -z
2010-12-21 14:43:44-0800 [Broker,client]   in dir C:\talos-slave\test\. (timeout 1200 secs)
2010-12-21 14:43:44-0800 [Broker,client]   watching logfiles {}
2010-12-21 14:43:44-0800 [Broker,client]   argv: ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n'
, '1', '-z']
2010-12-21 14:43:44-0800 [Broker,client]  environment: {'TMP': 'C:\\DOCUME~1\\cltbld\\LOCALS~1\\Temp', 'COMPUTERNAME': 'TALOS-R3-XP-003', 'MOZ_N
O_REMOTE': '1', 'USERDOMAIN': 'TALOS-R3-XP-003', 'TACFILE': '"c:\\talos-slave\\buildbot.tac"
                                           ', 'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files', 'PROCESSOR_IDENTIFIER': 'x86 Family 6 Mod
el 23 Stepping 10, GenuineIntel', 'PROGRAMFILES': 'C:\\Program Files', 'PROCESSOR_REVISION': '170a', 'SYSTEMROOT': 'C:\\WINDOWS', 'PATH': 'C:\\m
ozilla-build;C:\\mozilla-build\\msys\\bin;C:\\mozilla-build\\msys\\local\\bin;C:\\mozilla-build\\Python25;C:\\mozilla-build\\Python25\\Scripts;C
:\\mozilla-build\\hg;C:\\mozilla-build\\7zip;C:\\mozilla-build\\upx203w;C:\\WINDOWS\\System32;C:\\WINDOWS;', 'NO_EM_RESTART': '1', 'BB_BUILDBOT'
: '"C:\\mozilla-build\\python25\\Scripts\\buildbot" ', 'XPCOM_DEBUG_BREAK': 'warn', 'TACSCRIPT': '"c:\\tools\\buildbot-helpers\\buildbot-tac.py"
                                                                               ', 'TEMP': 'C:\\DOCUME~1\\cltbld\\LOCALS~1\\Temp', 'PROCESSOR_ARC
HITECTURE': 'x86', 'ALLUSERSPROFILE': 'C:\\Documents and Settings\\All Users', 'SESSIONNAME': 'Console', 'HOMEPATH': '\\Documents and Settings\\
cltbld', 'USERNAME': 'cltbld', 'LOGONSERVER': '\\\\TALOS-R3-XP-003', 'PROMPT': '$P$G', 'COMSPEC': 'C:\\WINDOWS\\system32\\cmd.exe', 'BOOTMODE':
'BKSTD', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH', 'PASSWORD': '"secret"
                                                                ', 'CLIENTNAME': 'Console', 'FP_NO_HOST_CHECK': 'NO', 'WINDIR': 'C:\\WINDOWS', '
HOMEDRIVE': 'C:', 'APPDATA': 'C:\\Documents and Settings\\cltbld\\Application Data', 'TYPE': '~0,5
                                                                  ', 'SYSTEMDRIVE': 'C:', 'HOSTNAME': 'talos-r3-xp-003
                                                     ', 'NUMBER_OF_PROCESSORS': '2', 'PWD': 'C:\\talos-slave\\test', 'PROCESSOR_LEVEL': '6', 'BB
_PYTHON': '"C:\\mozilla-build\\python25\\Scripts\\..\\python"', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'CONTROLFILE': '"c:\\buildbot-tac.control"
                                                                                             ', 'OS': 'Windows_NT', 'USERPROFILE': 'C:\\Document
s and Settings\\cltbld'}
2010-12-21 14:43:44-0800 [Broker,client]   closing stdin
2010-12-21 14:43:44-0800 [Broker,client]   using PTY: False
2010-12-21 14:44:14-0800 [-] Received SIGBREAK, shutting down.
2010-12-21 14:44:14-0800 [-] stopCommand: halting current command <buildbot.slave.commands.base.SlaveShellCommand instance at 0x01359F08>
2010-12-21 14:44:14-0800 [-] command interrupted, killing pid 2744
2010-12-21 14:44:14-0800 [-] trying process.signalProcess('KILL')

Comment 15

[Armen [:armenzg]]

Attachment: [screenshot] cmd.exe - Application Error - The application failed to initialize properly (0xc0000142).

Comment 16

[Armen [:armenzg]]

This is what happens:
* after we add attachment 499149 [details] [diff] [review] the unit test *step* can be aborted with "stop build" (which is not possible before)
* therefore, with the patch, we can reach the reboot step which does not finish

Anyone has any thoughts on why this could be happening?

I won't be able to deploy anything this week and will have to wait until I am back on January.

Tomorrow I will put my focus on the reboot issue.

Here is the code that gets called:
http://hg.mozilla.org/build/tools/file/tip/buildfarm/maintenance/count_and_reboot.py

By the end of tomorrow I will put xp-003 and xp-004 in production by reverting first attachment 499149 [details] [diff] [review] from them.

Comment 17

[Armen [:armenzg]]

Attachment: [tools] force reboot and wait 0 seconds to start reboot

This avoids the preemption of a reboot like on screenshot from attachment 499341 [details] to happen.

What are your thoughts on it?
This has been working on staging; can you think of any problems that this could introduce?

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 499589 [details] [diff] [review]
[tools] force reboot and wait 0 seconds to start reboot

I think some things (eg, debugger dialogs) could still hang the system up, but if this helps us avoid the hangups you're seeing, great!

Comment 19

[Chris Cooper [:coop] (he/him)]

Comment on attachment 499589 [details] [diff] [review]
[tools] force reboot and wait 0 seconds to start reboot

Sorry, thought I had reviewed this before I left.

Comment 20

[Armen [:armenzg]]

Comment on attachment 499589 [details] [diff] [review]
[tools] force reboot and wait 0 seconds to start reboot

This worked on staging. Let's take it.

Comment 21

[Armen [:armenzg]]

Comment on attachment 499589 [details] [diff] [review]
[tools] force reboot and wait 0 seconds to start reboot

http://hg.mozilla.org/build/tools/rev/c342e0b44940

This change will be picked up immediately by the following platforms:
* w2k3, xp, win7 & win7x64

Comment 22

[Armen [:armenzg]]

Attachment: [buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH

I found the root cause of XP machines not being able to kill unit test jobs.
It wasn't just the twisted patch but XP in itself with the PATH that we set for unit test jobs cannot kill the jobs because taskkill.exe cannot find framedyn.dll.

The reboot patch allowed us to reboot (disregarding any messages) but looking closer to the output of the runs they were not really orange as I thought.
They turned orange because they timed out and they timed out because XP with the PATH given would be unable to reach framedyn.dll.
After the timeout happened buildbot would be able to kill the step because of the twisted patch.

This patch solves the root cause.

Without including Wbem in the PATH I can't even run the jobs manually.

Here is the article by Microsoft explaining that Wbem has to be in the PATH:
http://support.microsoft.com/kb/319114

These are the steps to reproduce manually:
> set PATH=C:\mozilla-build;C:\mozilla-build\msys\bin;C:\mozilla-build\msys\local\bin;C:\mozilla-build\Python25;C:\mozilla-build\Python25\Scripts;C:\mozilla-build\hg;C:\mozilla-build\7zip;C:\mozilla-build\upx203w;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32;C:\WINDOWS;C:\WINDOWS\System32\Wbem
> wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1294139437/firefox-4.0b9pre.en-US.win32.zip
> unzip -o firefox-4.0b9pre.en-US.win32.zip
> wget --progress=dot:mega -N http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1294139437/firefox-4.0b9pre.en-US.win32.tests.zip
> unzip -o firefox-4.0b9pre.en-US.win32.tests.zip bin* certs* mochitest*
> python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --certificate-path=certs --autorun --close-when-done --console-level=INFO --symbols-path=symbols --chrome

Comment 23

[Chris Cooper [:coop] (he/him)]

Comment on attachment 501130 [details] [diff] [review]
[buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH

Good find. Do we need/want to revert the other patch?

Comment 24

[Armen [:armenzg]]

(In reply to comment #23)
> Comment on attachment 501130 [details] [diff] [review]
> include C:\Windows\System32\wbem into unit tests jobs' PATH
> 
> Good find. Do we need/want to revert the other patch?

No, we don't want to revert it.
This patch only avoids us having to deploy the twisted patch.

The reboot patch deals with the "cmd.exe - Application Error" attachment 499341 [details] which is different than the "taskkill.exe - cannot find framedyn.dll".

Comment 25

[Armen [:armenzg]]

Attachment: [buildbot-configs] enable XP optimized unit tests

With attachment 501130 [details] [diff] [review] and this patch we are ready to enable XP optimized unit tests.

Comment 26

[Armen [:armenzg]]

Comment on attachment 501130 [details] [diff] [review]
[buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH

checked-in to "default":
http://hg.mozilla.org/build/buildbotcustom/rev/67d99371f50f

Comment 27

[Armen [:armenzg]]

Comment on attachment 501130 [details] [diff] [review]
[buildbotcustom] include C:\Windows\System32\wbem into unit tests jobs' PATH

Merged to production-0.8:
http://hg.mozilla.org/build/buildbotcustom/rev/164c9740d9af

Comment 28

[Armen [:armenzg]]

Comment on attachment 501318 [details] [diff] [review]
[buildbot-configs] enable XP optimized unit tests

Landed on:
"default" - http://hg.mozilla.org/build/buildbot-configs/rev/8dfecf4da9f1
"production" - http://hg.mozilla.org/build/buildbot-configs/rev/c75c494415bf

Comment 29

[Armen [:armenzg]]

Reconfigured scheduler masters and test masters.
buildbot-configs: 164c9740d9af
buildbotcustom:   c75c494415bf

Comment 30

[Armen [:armenzg]]

I have announced this on the mailing lists.

What is left on this bug?
* file the perma-oranges we find
* see what is needed to enable the debug unit tests


We have to deploy the debug CRT (https://bugzilla.mozilla.org/show_bug.cgi?id=562459#c29) to be able to run debug unit tests.

The problem is that currently we have 20 slaves that can't receive packages through OPSI (bug 620517).
We could deploy this manually but I might have to end up be unlucky and have to deal with it :'(

Those 20 slaves also have the problem that it takes them a lot of time to login.
This occurs because it has a timeout for trying to reach the network device that they can't reach. This means that those slaves will always take them longer to come back online.
Adding more load with debug unit tests could increase the wait times if it takes them that long to come back online. We'll see.

Comment 31

[Phil Ringnalda (:philor)]

Reftest failures are all filed, everything but reftest is green (or known intermittent orange), and ready for you to unhide it.

Comment 32

[Armen [:armenzg]]

For copy/paste into wiki pages purposes let me re-write the perma-oranges filed:

PERMA-ORANGES (all of them reftests):
* bug 623405 - anim-text-rotate-01.svg & dynamic-text-04.svg
* bug 623423 - editor/xul/empty-1.xul, editor/xul/autocomplete-1.xul, editor/xul/emptyautocomplete-1.xul, editor/xul/number-3.xul, editor/xul/number-5.xul, editor/xul/numberwithvalue-1.xul
* bug 623454 - reftests/bugs/580863-1.html
* bug 623450 - layout/reftests/bugs/579323-1.html
** bug 623452 - layout/reftests/bugs/579985-1.html (depends on bug 623450)
** bug 623456 - layout/reftests/bugs/581317-1.html (depends on bug 623450)
** bug 623460 - reftests/ogg-video/encoded-aspect-ratio-1.html (depends on bug 623450)
** bug 623403 - gradient-live-01*.svg (depends on bug 623450)

Already fixed:
* bug 623459

Comment 33

[Armen [:armenzg]]

Attachment: [screenshot] taskkill.exe cannot find framedyn.dll

I never uploaded this screenshot.

It shows how a "taskkill.exe cannot find framedyn.dll" prompt appears and blocks buildbot from finishing that step.

attachment 501130 [details] [diff] [review] fixes this problem.
That is, buildbot needs to set for each step 'C:\\WINDOWS\\System32\\Wbem' as part of the PATH. In case taskkill.exe has to be called for that step to kill any processes that the step might have created (IIUC).

Here is the article by Microsoft explaining that Wbem has to be in the PATH:
http://support.microsoft.com/kb/319114

Comment 34

[Armen [:armenzg]]

STATUS UPDATE:
* all test suites for XP are now green
* philor has disabled w2k3 from the tbox pages (he should be announcing soon)

Comment 35

[Armen [:armenzg]]

Resuming this work.

Let's add debug unit tests for XP.

Comment 36

[Armen [:armenzg]]

I am having trouble getting the XP debug unit tests to run.
Luckily ted will be able to give me a hand with this early next week.

Comment 37

[Armen [:armenzg]]

Yay! The XP debug unit test run on staging went through and we only have 3 test suites that contain perma-oranges.

TODO: File bugs for these oranges.

Rev3 WINNT 5.1 mozilla-central debug test mochitest-other: 
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299276935.1299283630.13684.gz
7558 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/layout/base/test/chrome/test_leaf_layers_partition_browser_window.xul | Leaf layers should form a non-overlapping partition of the browser window

Rev3 WINNT 5.1 mozilla-central debug test reftest: 
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299283826.1299286283.25568.gz
REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-01.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-02.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-03.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-04.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-05.svg | image comparison (==)
REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/tspan-rotate-06.svg | image comparison (==)

Rev3 WINNT 5.1 mozilla-central debug test mochitests-2/5:
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299273603.1299275641.14918.gz&fulltext=1
6366 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/scriptaculous/test_Scriptaculous.html | testInPlaceEditor - 31 assertions, 2 failures, 0 errors
6367 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/tests/mochitest/ajax/scriptaculous/test_Scriptaculous.html | testInPlaceEditor - 31 assertions, 2 failures, 0 errors

Comment 38

[Armen [:armenzg]]

I run it a second time.

(In reply to comment #37)
> Rev3 WINNT 5.1 mozilla-central debug test mochitest-other: 
> http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299276935.1299283630.13684.gz
> 7558 ERROR TEST-UNEXPECTED-FAIL |
> chrome://mochitests/content/chrome/layout/base/test/chrome/test_leaf_layers_partition_browser_window.xul
> | Leaf layers should form a non-overlapping partition of the browser window
> 
This is the only one failure that has repeated from the previous run.
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1299543394.1299546049.14379.gz

Comment 39

[Armen [:armenzg]]

Attachment: [buildbot-configs] enable XP debug unit tests

Comment 40

[Armen [:armenzg]]

Comment on attachment 517753 [details] [diff] [review]
[buildbot-configs] enable XP debug unit tests

checked-in to default without the code from another patch.
http://hg.mozilla.org/build/buildbot-configs/rev/9dd3078a9ac4

This will be picked up with Thursday morning's scheduled reconfig.

Comment 41

[Armen [:armenzg]]

This got enabled today.
And the suites are looking green! (except M1 & Mo)

http://hg.mozilla.org/build/buildbot-configs/rev/f1a4d01c8caa

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Is it known that Mo is hanging in mochitest-browser-chrome ? It generates 50M of log, at which point we clamp the output, but the job continues at least 20 hours. Lots of js stacks from Places in the logs I think.

Comment 43

[Phil Ringnalda (:philor)]

I'd expect that to be (did expect that it was, in the logs I glanced at) ye olde bug 631447 - does that do the same things when it hits non-XP?

Comment 44

[Phil Ringnalda (:philor)]

Sort of difficult to tell frequency, since they also expanded bug 629610 into browser_focus_steal_from_chrome.js, which kills the run before bug 631447 has a chance to overflow the log. Between them, those two are not actually quite permanent, though, since http://tbpl.mozilla.org/?tree=MozillaTry&noignore=1&rev=a1cb76b70af6 did manage to succeed in browser-chrome (the only thing on m-c or try which has, so far).

Comment 45

[Armen [:armenzg]]

For Mo we have:
- bug 640889 - chrome -> 7558 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/layout/base/test/chrome/test_leaf_layers_partition_browser_window.xul | Leaf layers should form a non-overlapping partition of the browser window
- no bug? - browser-chrome -> TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/dom/tests/browser/browser_focus_steal_from_chrome_during_mousedown.js | Exited with code -1073741819 during test run

For M1 we have (which is green on the last changeset):
- bug 640887 36170 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Test expected to fail, but passed: conformance/canvas-test.html

I am thinking of disabling browser-chrome for Windows XP until we figure it out on staging.
philor what do you say? I know it is intermittent but quite frequent and the slaves get stuck until someone reboots the machine. Is this right nthomas?

We would also need the twistd patch at some point to be able to stop the jobs that get into a similar state.

Comment 46

[Armen [:armenzg]]

I am rebooting XP machines that are getting stuck with mochitest-other jobs.

I will put a patch as soon as I can.

Comment 47

[Phil Ringnalda (:philor)]

Seems reasonable to me - mak has a handle on it, but it's not a test-only patch so we're stuck waiting for the tree to reopen (and then we'd still be stuck waiting for people to merge it around to every tree).