Monitoring a Log File
From CodeCodex
There is frequently a need to scan system or application log files for notifications of particular events and the like. There are some subtle points about reading from a file that another process is actively writing to:
- When you reach the current end of the file, some more content may be appended a few seconds or a few minutes later. Thus, you need to wait a while and then try to resume reading from the current position. This requires clearing the EOF indication without closing and reopening the file.
- Many systems implement logfile rollover, where periodically the current log file is closed and renamed, and a new log file opened with the previous name. This is to keep the log files from growing without bounds.
Some Python code that handles both these issues is as follows:
LogFileName = "/var/log/messages" # or whatever log file you're wanting to monitor
MonitorInterval = 60 # number of seconds between checks for new content
LogFile = None # to begin with
while True :
if LogFile == None :
LogFile = open(LogFileName, "r")
#end if
while True :
Line = LogFile.readline()
if Line == "" :
break # end of file reached
#end if
... do whatever processing of Line is necessary ...
#end while
if \
os.path.realpath("/proc/self/fd/%d" % LogFile.fileno()) \
!= \
os.path.realpath(LogFileName) \
:
# rollover has started, might still be in progress
LogFile.close()
LogFile = None
else :
LogFile.seek(0, 1) # clear EOF state
#end if
time.sleep(MonitorInterval)
# wait for more content, also gives time for rollover to complete
#end while
Note that the check whether the log file still has the same name is reliant on the functionality of procfs under Linux—will this work under any other systems?
