Data save All taking Long time because of persistent feeders

[guru1988fortm1]

Hi All,

I am using TM1 9.5.2 with cognos 10 BI security

we have enabled the data save all chore that is scheduled to run for every 3 hours with Persistent Feeders=T
(data save all is present in the epilog of an separate process which has got nothing else)

Sometimes , during peak hours, our users will have bulk of data loads and entries,

During those times data save all takes a long time around 30-35 minutes to save even a data of 2 GB. Last week, the system crashed after save data all with a statement
'feeder date is older than the cube date' (really could nt understand what it means)

i really dont understand this persistent feeders, it is for the faster restart of the system when the server crashes or when we want to restart during the peak hours

But everytime i restart the server, it deletes the feeders and creates them again and it takes the same time of 2 hours and

These feeders files occupy an total space of 40 GB in the disk

So, is it good to disable the persistent feeders?

Can anyone suggest your opinions and help me on this

Thanks
Regards
Guru

[whitej_d]

When you restart the server, it should load the feeders straight into memory. The fact that it deletes them when you restart suggests something not quite right.

TM1 would delete the feeders files and re-evaluate based on the timestamp of the .feeders file compared to the timestamp of the corresponding .cub file. (Although this may have been different in 9.5.2). If the .cub file is newer than the .feeders file it assumes the feeders are invalid and reprocesses them all on start-up.

It sounds like you have a pretty big system to have 40GB of feeders files. If you don't give it plenty of time to shut down (for example if you reboot the machine instead of stopping the TM1 service first) there's a chance it isn't saving the feeders files fully and then rejecting them on start-up.

It's still always going to take a long time for a SaveDataAll with feeders that big, so if it's causing you issues during the day, you may prefer to disable persistent feeders. This should significantly speed up the SaveData and it won't affect performance of the system during use, but you won't be able to get the fast start-up benefits (which you're not getting anyway!)

[guru1988fortm1]

Thanks Whitej_d

It sounds like i should have my system without persisten feeders

But is this all about for persistent feeders, only for the rapid server restart?

Also, doing this will reduce the size of my TM1 database from 40 GB to lesser size or its gonna be the same

Thanks
Regards
Guru

[tomok]

The only purpose of Persistent Feeders is so that the TM1 service can restart faster. That's because the .feeders file contains all the intersections that should be fed and thus TM1 doesn't have to re-evaluate the feeders while the service is coming up. If your system is such that a lot of data is loading and/or changed during the working day, thus requiring a lot of live feeder re-evaluation, then Persistent Feeers may not make sense. All you are doing is swapping time. You can either spend the time creating the .feeders files (during SaveDataAll) and save time on startup, or spend the time on startup and have faster SaveDataAll execution.

Not having persistent feeders may save a little disk space but it will have no effect on the size of your model.

[Gabor]

10.1.1 FP2 and 10.2 FP1 seem to have finally solved a bunch of Persistent Feeder issues, which were "built in" since this feature was launched with 9.5.1.

All versions before showed random issues with detecting a (non-existing) mismatch between .feeder and .cub timestamps.
Using more than 1 DB folder in tm1s.cfg turned this fact even from bad to worse.
All of that in relation to specific datamodels of course, what made it even harder to make IBM aware of it.

Another thought as your amount of feeding is pretty high.
Are we talking about unique information in the feeder destination, or is source information duplicated by feeding into consolidations for example?

There are "advanced" techniques in TM1 to reduce feeding or get rid of it for certain calculations.
I have seen 10 GB to go down to 100 MB if your architect is aware of how to do it.
One common example is to make a traditional calculated N-element a consolidation and instead feeding to place the feeder source as child under it.
This enables so called feeding via natural consolidation.

[lotsaram]

Regardless of whether the feeders are correct or not I would also STRONGLY recommend upgrade to 10.1 FP2 or 10.2 FP1 as from v10.1 issues with conditional feeders on startup seem to have finally been fixed so you can benefit from the quick model startup. AND maybe more importantly in this case the locking during SaveDataAll is basically non-existent, so even is the data save operation takes a long time it won't affect users.

[guru1988fortm1]

Thanks all for your comments

I have made the persistent feeders=F and the server seems to be working fine , The data save all is also faster

Even, i have recommended for an migration to 10.1

This forum always gives me a best solution