Announcement

Collapse
No announcement yet.

Viewing certain events crashes Hyena's event viewer

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Viewing certain events crashes Hyena's event viewer

    Today our mail admin complained that his installation of Hyena tends to crash when he uses it to view system event logs, terminating with the "Hyena v10.5.3 stopped working" message box. Together we could narrow down the problem to a very specific cause. The crash only occurs when he tries to view a certain event generated by the Sophos Antivirus Scanner that runs on our servers. Maybe the error message is somehow malformed and crashes the event viewer. I cross checked that with other installations and could reproduce the same problem on a Win 7 client and a Win 8.1 (both 64 bit German) client. Hyena versions are 10.5 D and E, 64 bit English. The problem does NOT occur with 32-bit installations of Hyena.

    Hyena doesn't seem to have a problem with other events either, only with this specific error message. The same log event could be read without problems using the windows event viewer, so at least I can post it here:

    Event Type: Error
    Event Source: SAVOnAccessControl
    Event Category: None
    Event ID: 17
    Date: 14.03.2014
    Time: 10:25:59
    User: N/A
    Computer: *****
    Description:
    The on-access driver failed to determine the new name for file %2 following a rename.


    Data:
    0000: 00 00 04 00 01 00 6e 00 ......n.
    0008: 00 00 00 00 11 00 3d e0 ......=à
    0010: 5e 00 3a 01 83 01 00 c0 ^.:.?..?
    0018: 00 00 00 00 00 00 00 00 ........
    0020: 00 00 00 00 00 00 00 00 ........
    0028: 01 00 00 c0 ...?

    There is something interesting about the error description above. When I view it with a 32-Bit version of Hyena it reads:

    The on-access driver failed to determine the new name for file (null) following a rename.

    So maybe the 64-bit versions of Hyena chokes on the (null) value in the description string.

  • #2
    When does the crash occur? Is it when displaying the in right-hand window, or when you double-click on the event? How do you get to this point to view the events (such as right-click Events > Filter Events, or Events > Event Viewer)?

    Comment


    • #3
      You have diagnosed your problem fairly well. I too suspect the crash is happening because there isn't a value for the file. But how this mechanism works is a bit hard to explain. The message string in the event log looks like: "The on-access driver failed to determine the new name for file %2 following a rename.". What MMC event view / Hyena have to do is get the value for %2 from the .dll file supplied by the event source (AV vendor). This .dll is not returning a filename, and probably causing the crash. If you notice on the event message from MMC, the %2 is left in the message, because the lookup failed.

      Unfortunately, Hyena uses a Windows function to do the parameter replacement, and the crash is probably happening there. We'll try to find some events on our end that have the "(null)" in the message, as that too is not coming from Hyena, so the Windows function (32-bit ?), must be doing that on its own. Perhaps the 64-bit Windows function just blows up.

      What you might try to do is to email the vendor (Sophos) and tell them that the event message string for events from "SAVOnAccessControl" for event ID 17 is missing parameter %2. Just show them the event details you have above. Maybe they have a fix for it in an updated message .dll file and everyone will be happy.

      I'll see if I can maybe repro this, but its hard because I have to find another event system with a similar bug in its message file.
      Kevin Stanush
      SystemTools Software Inc.

      Comment


      • #4
        I view the events either by right-click Events > Filter events > System or by expanding the events node and clicking on System. I can see all events in the right-hand window, only when I double-click on one of the events mentioned above Hyena terminates with the "Hyena v10.5.3 stopped working" message box.


        Events > Event Viewer launches the Windows event viewer. There I have no trouble to view the event.
        When Hyena terminates I see this in my local event log:


        Log Name: Application
        Source: Application Error
        Date: 14.03.2014 16:10:27
        Event ID: 1000
        Task Category: (100)
        Level: Error
        Keywords: Classic
        User: N/A
        Computer: ******.******.at
        Description:
        Faulting application name: Hyena_x64.exe, version: 10.5.4.0, time stamp: 0x531e0a42
        Faulting module name: ntdll.dll, version: 6.3.9600.16502, time stamp: 0x52c359e8
        Exception code: 0xc000041d
        Fault offset: 0x000000000008e4c3
        Faulting process id: 0x27e0
        Faulting application start time: 0x01cf3f977263dd68
        Faulting application path: C:\Program Files\Hyena\Hyena_x64.exe
        Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
        Report Id: c61270dc-ab8a-11e3-beb9-00a0c6000000
        Faulting package full name:
        Faulting package-relative application ID:
        Event Xml:
        <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
        <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2014-03-14T15:10:27.000000000Z" />
        <EventRecordID>133794</EventRecordID>
        <Channel>Application</Channel>
        <Computer>******.******.at</Computer>
        <Security />
        </System>
        <EventData>
        <Data>Hyena_x64.exe</Data>
        <Data>10.5.4.0</Data>
        <Data>531e0a42</Data>
        <Data>ntdll.dll</Data>
        <Data>6.3.9600.16502</Data>
        <Data>52c359e8</Data>
        <Data>c000041d</Data>
        <Data>000000000008e4c3</Data>
        <Data>27e0</Data>
        <Data>01cf3f977263dd68</Data>
        <Data>C:\Program Files\Hyena\Hyena_x64.exe</Data>
        <Data>C:\WINDOWS\SYSTEM32\ntdll.dll</Data>
        <Data>c61270dc-ab8a-11e3-beb9-00a0c6000000</Data>
        <Data>
        </Data>
        <Data>
        </Data>
        </EventData>
        </Event>

        Comment


        • #5
          Maybe this additional information is helpful: when I use the windows event viewer I always get the "%2" string, on machines with 32-bit-Windows as well as on machines with 64-bit-Windows. I have installed both versions of Hyena on 64-bit-Windows clients and I see the "(null)" string only when I use 32-bit-Hyena.


          Will try to open a case at Sophos but their support is QUITE sluggish compared to your's ;-) They are still fixing a problem that popped up in December...

          Comment


          • #6
            Thanks for the additional info. I forgot to ask what exact steps are taken in Hyena to generate the crash. This matters, because there are two different event sub-systems in Windows, and I need to know which one is being used. If you are looking at the server in Hyena's left window, and then expand the events, and find the event log (System), double-click it, and view the events in the right window, then double-click on one of these events, at that point you can get the crash ? If you are looking at one of these events (the details) that does not crash, does the event details dialog have a tab at the top labeled "General" ?
            Kevin Stanush
            SystemTools Software Inc.

            Comment


            • #7
              Sorry about the multiple quick followups, but since you were in Austria, and I wanted to try to get back to you as quickly as possible. There is an option in Hyena to disable event parameter replacement, which is mainly there to speed things up and help debug some event message issues. Parameters begin with double percent symbols (%%), but just in case this is where the problem is, you can see if this works around the issue which gives us a new place to look:

              Go to Tools > Settings > Advanced, and find the setting for "ReplaceEventLogParamters", and set it to "FALSE".

              Then, try to view one of the events that causes the crash.

              It should still crash, but I want to make sure the event message parameter is not something like %%4354, and then that gets replaced with %2, which in turn causes the crash. What we are trying to do is get a look at the raw event message string before it gets pre-processed.

              Thanks for your help.
              Kevin Stanush
              SystemTools Software Inc.

              Comment


              • #8
                If you are looking at the server in Hyena's left window, and then expand the events, and find the event log (System), double-click it, and view the events in the right window, then double-click on one of these events, at that point you can get the crash ?

                Yes, these are the exact steps to trigger the error.

                If you are looking at one of these events (the details) that does not crash, does the event details dialog have a tab at the top labeled "General" ?

                No tab there, I have enclosed a screenshot to show what the dialog looks like.

                Go to Tools > Settings > Advanced, and find the setting for "ReplaceEventLogParamters", and set it to "FALSE".

                "FALSE" seems to be the default setting. I changed it to "TRUE" but this did not make a difference.
                Attached Files

                Comment


                • #9
                  OK, that tells us what event sub-system is giving the error. Let us know what the operating system is on the client and server as the newer event reading system in Hyena uses a different GUI entirely and different protocols.

                  See if you can enable the newer event log reading system by selecting Tools > Settings > Advanced, and near the bottom of the list, find the setting for "EnableCrimsonLogs" and set it to 'True'. Restart Hyena and view the log by just double-clicking it. You will notice a different GUI with a larger dialog. See if reading this event this way does not cause a crash. And, what is displayed for the message as well.

                  Thanks again for your help.
                  Kevin Stanush
                  SystemTools Software Inc.

                  Comment


                  • #10
                    I admit I am a bit confused now. I checked the advanced settings you mentioned and noticed that "EnableCrimsonLogs" was already set to true. When I double-click on any (non-crashing) events in the right window I get exactly the dialog I have attached above. There is no different GUI or larger dialog.
                    I have verified this on my Workstation (64-Bit-Windows 8.1 German, 64-bit-Hyena 10.5.E) and on a server (64-Bit-Windows 2008R2, 64-bit-Hyena 10.5.E)

                    Comment


                    • #11
                      I too am a bit confused at this point. Given the version that you have and your settings, and the versions of your client and server, you should be able to use the newer event log system (called "Crimson" by Microsoft). If you just expand the server in the left window, go to Events, and expand that, it should look like this when viewing Crimson logs:
                      events.jpg
                      The main way to visually tell this event log view from the older view is the presence of the "Microsoft" folder at the bottom.

                      When you double-click on one of these logs, the right window columns will be a bit different, particularily having a column labeled "channel". Double-clicking on one of these events will display a larger dialog than what you attached above.

                      While Hyena's event filtering still uses the older event logic which is where the crash is happening, I'm concerning why your system is not using the new "crimson" logs.

                      So, let me know if your event log view in the left window looks like my image, basically I want to know if you see the "Microsoft" folder or not.

                      We tried to find an event on our systems to try to reproduce this issue, but can't find one with a missing parameter string like that.

                      But we did find a problem with the 32-bit version that while it should not affect this particular problem, I may try to get you a patched German 64-bit version to try just to see if it fixes the crash.

                      However, if we can figure out why you are not reading the Crimson logs in the first place, we can get this corrected a different way.

                      Sorry for the delay in coming up with something.
                      Kevin Stanush
                      SystemTools Software Inc.

                      Comment


                      • #12
                        I now can see the reason for our bafflement. I had experienced the crashes with versions of Hyena installed on my client and on two different servers we use for administrative purposes. I thought you referred to this information when you said "Let us know what the operating system is on the client and server... "


                        The operating system on the servers that give us these troubles is Win 2003R2. This was the information you actually wanted to know. I instantly noticed the GUI you mentioned when I connected to one of our more recent servers. I tend to use the builtin event viewer of Windows, so I had never looked at the "Crimson" logs with Hyena before.

                        So unfortunately these crashes occur on servers that don't have "Crimson" logs.
                        Sorry for the confusion.

                        Comment


                        • #13
                          No problem, the problem is where I assumed it should be. The only trick is trying to duplicate it. One thing I'd like to try is to force this same error on our end, but to do that, we would need to know what Sophos product you are using so that we can determine if we can install it. Try to go to www.sophos.com and point me to the product page. For example, is it this one ? :

                          http://www.sophos.com/en-us/products...-security.aspx

                          Then, assuming its a product that we can install, I'll need to know if you know what generated that event log message. Is there something that happens that causes that message ? Even if you don't know, I can generate my own message, but first we need to try to install the Sophos product (free eval...)
                          Kevin Stanush
                          SystemTools Software Inc.

                          Comment


                          • #14
                            According to our Security Team the installed product is called "Enduser Protection Suite". So I guess you have to select the free trial for "Enduser Protection" at http://www.sophos.com/en-us/products/enduser-protection-suites/free-trial.aspx


                            We get these error entries on our W2003R2-Terminal servers only if the DFSR-feature is installed. We use that to replicate lots of small configuration files from a central server to our branch offices. It seems that Sophos cannot keep up fast enough with all the staging and renaming that is going on so it tries to scan files that are no longer here, hence leading to these errors.

                            Comment


                            • #15
                              Thanks for providing that link. With it, I was able to download the product, install it, force in an event like the one you got, and duplicate the error. Even better, we found a way to prevent the crash when a mal-formed event message is found, and just show the raw event message, ie "The on-access driver failed to determine the new name for file %2 following a rename". We still have to do some additional testing to see how everything performs, but should be able to get you a patch by Monday morning.

                              Thanks again for your help on this one.
                              Kevin Stanush
                              SystemTools Software Inc.

                              Comment

                              Working...
                              X