TM1 Log Files - SOX Requirements

[wkeller]

We are working to implement audit and SOX controls in our TM1 v10.1 system via existing TM1 log files, but we are encountering several challenges - as follows:

Generating Log Reports:
- Audit Log: Unable to export more than 100 lines of data at a time (unacceptable)
- Transaction Log: Unable to copy/paste more than two days of transactions without crashing Excel (due to large volume of data)

Other Issues:
- TM1 Service must be stopped to access tm1s.log - is there an alternate method to get this file/data?
- Analysis of large volume of data is very challenging - need to identify certain types of events, such as security updates, data updates, outages, Admin activity, etc
- Ability to auto-generate log files via TI Process?


Any assistance/advice from someone addressing with similar issues is appreciated!

Thanks!

[Carl Lonsdale]

You might look into scripting or pulling the Log files directly.
Audit Log files are stored in the logging directory (Tm1Data by default):
Audit Logs: tm1rawstore.(Timestamp) (I believe?)
Transaction Logs: tm1s(timestamp).log

You cannot access the tm1s.log file while the instance is running because this is the active transaction log (the server will also try and recover state from this file in the event of a crash). There are currently no (Good) work-arounds for this.
Also you will run into a similar issue with the current Audit File (tm1rawstore). The current file will also be write protected while the instance is up.

In Terms of handling Large Data volumes, Linux commands are very useful for this (GREP|GREP > relevant log, etc) and scripts to this effect can parse HUGE files for only relevant data entries very quickly. There are also some large Volume file read utilities available: UltraEdit for example.

Lastly, can your processes auto generate logs, Yes. You can write relevant data to output as necessary using the ASCIIOUTPUT command.
As Tm1 also accepts text files as input and logs are text files, you can also write TI scripts to scan the logs for you and output only relevant entries to a separate log file, or even scan the log files to say: output a list of process
-> subprocess calls. Although other scripting languages are much better suited to do this. (TI process to call a script to parse logs?)
Hope this helps!

[David Usherwood]

You cannot access the tm1s.log file while the instance is running because this is the active transaction log (the server will also try and recover state from this file in the event of a crash). There are currently no (Good) work-arounds for this.

Executing SaveDataAll will close tm1s.log rename it to tm1s<timestamp>.log and open a new one - and the entries you have up to that point are now available.
Also, the audit logs (as opposed to the transaction logs, which the above is about) are somewhat ugly but can be parsed, either with grep-type tools or in TI itself. You set an interval in the config file which can be quite short, and you can flush the rawstore manually:

AuditLogUpdateInterval=60

[TableManagerOne]

The documentation looks a bit wonky, but you can access the live transaction log via the C API (TM1ServerLogOpen, TM1ServerLogNext, and TM1ServerLogClose.) Probably what Architect uses for its viewer.

[Alan Kirk]

The documentation looks a bit wonky, but you can access the live transaction log via the C API (TM1ServerLogOpen, TM1ServerLogNext, and TM1ServerLogClose.) Probably what Architect uses for its viewer.

TableManagerOne speaks the truth, but I'd add one caution; I've always found that particular API call to be painfully slow (which is not in itself a problem) but also that the log file access tends to slow the living bejebbers out of the server (which is). Though I'll admit that I haven't tried it on any of the 10.x series. For that reason my preference when I need to access the logs is to go David's data save route, which releases TM1's grip over the saved transaction logs and I can open those and grab what I need to my heart's content.

Obviously that's not a solution if you need effectively real time access to the transaction logs, but I'd be wary about doing anything that required that anyway. You run the risk of performance hits in such a case.

[wkeller]

I sincerely appreciate each of your responses to my logging inquiry! I did not have much success with internet searches on this topic (beyond IBM documentation) previously.I knew that if any addiional knowledge/expertise existed, this forum would be the most reliable souce of information.

In regards to DataSaveAll, we currently perform Saves every two hours (to minimize data loss risk - yes, this is probably extreme) which results in alot of small tm1s<timestamp>.log files being generated. We need a log file for each month. As a result, our current plan is to stop the service at the end of each month to allow us to grab a copy of the master tm1s.log file (3/31 will be our first attempt) for analysis.

In regards to parsing the various log file data, our company will be utilizing a tool called 'Q RADAR' to gather/filter the massive amounts of information to isolate meaningful audit related activity. However; I will forward them the recommendations for Linux and UltraEdit. Many our our TI Processes do write some info to a Processes 'Control' cube - perhaps we can expand the degree of detail provided.

TI scripting is another solid option we may need to explore as an alternative to the above methods - but may require additional research to develop.

We will be reviewing the tm1s.log, tm1auditstore.log and tm1server.log files on a monthly basis for the info we need for audit requirements.

I'll update this thread with our results for furure reference.

Thanks again!

Bill Keller

[tomok]

We need a log file for each month. As a result, our current plan is to stop the service at the end of each month to allow us to grab a copy of the master tm1s.log file (3/31 will be our first attempt) for analysis.

What "master tm1s.log file"? There is no such thing. Each time you either 1) issue a SaveDataAll, or 2) take down the service, the contents of whatever is in the tm1s.log file gets saved to the timestamp version and then a new empty tm1s.log file is generated. If you want a log file for the entire month then you'll need to concatenate the contents of all the timestamped files for that month. This is really easy to do via several different command lines like TYPE or COPY.