personal/weatherlog: weatherlog/logger.py annotate

annotate weatherlog/logger.py @ 14:c01f9929ae38

Make logger and HTTP writer more general and resilient. This makes the logger and HTTP writer more general, by removing any dependency upon the exact data type they are writing. They can now handle any type of BSON-serializable dict, and track what they have sent by keeping track of the last *byte*, not the last timestamp.

author	Paul Fisher <paul@pfish.zone>
date	Tue, 15 Oct 2019 22:40:24 -0400
parents	8a350ec1aa78
children	770215590d80

rev	line source
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	1 """Module to log temperatures to a directory, and an external website."""
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	2
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	3 import abc
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	4 import collections
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	5 import datetime
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	6 import fcntl
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	7 import os
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	8 import pathlib
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	9 import threading
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	10 import time
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	11 import typing as t
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	12
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	13 import attr
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	14 import bson
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	15 import pytz
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	16
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	17 BSON_FILENAME = "temps.bson"
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	18 OLD_LAST_TS = "last-sent"
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	19 START_BYTE = "start-byte"
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	20
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	21
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	22 class RemoteWriter(metaclass=abc.ABCMeta):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	23
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	24 BATCH_SIZE = 1000
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	25
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	26 @abc.abstractmethod
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	27 def write(self, readings: t.Iterable[t.Dict[str, object]]) -> None:
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	28 raise NotImplementedError()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	29
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	30
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	31 class RemoteWriteError(Exception):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	32 """Error to be raised by RemoteWriter.write."""
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	33 pass
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	34
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	35
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	36 @attr.s(auto_attribs=True, frozen=True, slots=True)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	37 class ReadingPosition:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	38 # The encoded reading that was written.
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	39 data: bytes
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	40
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	41 # The index of the byte immediately following this one.
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	42 end: int
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	43
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	44
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	45 class BufferedLogger:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	46 """A resilient logger which logs to a local file and a RemoteWriter."""
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	47
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	48 WAIT_TIME = 2
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	49
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	50 def __init__(self, directory: str, writer: RemoteWriter):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	51 self._writer = writer
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	52 self._path = pathlib.Path(directory)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	53 self._file = _open_exclusive(self._path / BSON_FILENAME)
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	54 self._old_last_sent = self._path / OLD_LAST_TS
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	55 self._start_byte = self._path / START_BYTE
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	56 unsent = _read_unsent_and_upgrade(
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	57 self._file, self._old_last_sent, self._start_byte)
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	58 self._send_queue = collections.deque(unsent)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	59 self._running = False
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	60 self._remote_thread: t.Optional[threading.Thread] = None
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	61
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	62 def start(self) -> None:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	63 """Starts the bit which logs to HTTP."""
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	64 self._running = True
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	65 self._remote_thread = threading.Thread(target=self._send_internal)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	66 self._remote_thread.start()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	67
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	68 def close(self) -> None:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	69 """Stops the logger, closes all files, and stops writing to HTTP."""
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	70 fcntl.flock(self._file, fcntl.LOCK_UN)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	71 self._file.close()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	72 self._running = False
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	73 if self._remote_thread:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	74 self._remote_thread.join()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	75
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	76 def write(self, reading: t.Dict[str, object]) -> None:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	77 encoded = bson_encode(reading)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	78 self._file.write(encoded)
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	79 self._file.flush()
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	80 byte = self._file.tell()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	81 self._send_queue.append(ReadingPosition(encoded, byte))
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	82
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	83 def _send_internal(self) -> None:
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	84 to_send: t.List[ReadingPosition] = []
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	85 while True:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	86 # Wait for multiple entries to build up in the queue.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	87 time.sleep(self.WAIT_TIME)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	88 while len(to_send) < self._writer.BATCH_SIZE:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	89 # Pop all the values we can off the queue.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	90 try:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	91 to_send.append(self._send_queue.popleft())
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	92 except IndexError:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	93 break
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	94 if not self._running:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	95 # Stop if we've been asked to stop.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	96 break
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	97 if not to_send:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	98 # If there's nothing to send, don't try to send anything.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	99 continue
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	100 try:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	101 # Try writing out the values.
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	102 self._writer.write(e.data for e in to_send)
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	103 except RemoteWriteError:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	104 pass # If it fails, just try again next time.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	105 else:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	106 # If we succeeded, record our success.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	107 last_sent = to_send[-1]
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	108 self._update_start_byte(last_sent.end)
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	109 to_send.clear()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	110
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	111 def _update_start_byte(self, byte: int) -> None:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	112 start_byte_name = self._path / (START_BYTE + ".new")
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	113 with start_byte_name.open('w') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	114 outfile.write(str(byte))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	115 start_byte_name.rename(self._start_byte)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	116
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	117
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	118 def _atomic_write(file: pathlib.Path, contents: str) -> None:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	119 """Writes a string to a file, atomically."""
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	120 new_name = file.with_name(file.name + '.new')
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	121 with new_name.open('w') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	122 outfile.write(contents)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	123 new_name.rename(file)
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	124
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	125
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	126 def _open_or_create(path: pathlib.Path) -> t.BinaryIO:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	127 while True:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	128 try:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	129 return path.open('r+b')
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	130 except FileNotFoundError:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	131 pass
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	132 try:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	133 return path.open('x+b')
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	134 except FileExistsError:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	135 pass
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	136
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	137
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	138 def _open_exclusive(path: pathlib.Path) -> t.BinaryIO:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	139 file = _open_or_create(path)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	140 try:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	141 fcntl.flock(file, fcntl.LOCK_EX \| fcntl.LOCK_NB)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	142 except BlockingIOError as ex:
6 8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	143 file.close()
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	144 raise OSError('Another copy of the logger is running.') from ex
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	145 return file
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	146
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	147
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	148 def _read_unsent_and_upgrade(
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	149 infile: t.BinaryIO,
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	150 last_sent_file: pathlib.Path,
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	151 start_byte_file: pathlib.Path,
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	152 ) -> t.List[ReadingPosition]:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	153 _maybe_upgrade_last_sent(infile, last_sent_file, start_byte_file)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	154 start_byte = _read_start_byte(start_byte_file)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	155 infile.seek(start_byte, os.SEEK_SET)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	156 reader = bson.decode_file_iter(infile, BSON_OPTIONS)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	157 readings: t.List[ReadingPosition] = []
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	158 end_pos = infile.tell()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	159 try:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	160 for entry in reader:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	161 data = bson_encode(entry)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	162 end_pos = infile.tell()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	163 readings.append(ReadingPosition(data, end_pos))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	164 except bson.InvalidBSON:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	165 infile.seek(end_pos, os.SEEK_SET)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	166 infile.truncate(end_pos)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	167 return readings
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	168
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	169
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	170 def _read_start_byte(path: pathlib.Path) -> int:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	171 try:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	172 with path.open('r') as infile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	173 return int(infile.read())
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	174 except (OSError, ValueError):
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	175 return 0
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	176
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	177
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	178 def _maybe_upgrade_last_sent(
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	179 infile: t.BinaryIO,
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	180 last_sent_file: pathlib.Path,
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	181 start_byte_file: pathlib.Path,
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	182 ) -> None:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	183 """If there's a last-sent file, upgrades it to start-byte."""
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	184 last_sent = _read_last_sent(last_sent_file)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	185 if not last_sent:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	186 return
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	187 reader = bson.decode_file_iter(infile, BSON_OPTIONS)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	188 last_good = infile.tell()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	189 try:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	190 for entry in reader:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	191 try:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	192 timestamp: datetime.datetime = entry['sample_time']
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	193 except KeyError:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	194 continue # Invalid entry; skip it.
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	195 if last_sent < timestamp:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	196 break
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	197 last_good = infile.tell()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	198 except bson.InvalidBSON:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	199 infile.seek(last_good, os.SEEK_SET)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	200 infile.truncate(last_good)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	201 _atomic_write(start_byte_file, str(last_good))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	202 last_sent_file.unlink()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	203
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	204
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	205 def _read_last_sent(path: pathlib.Path) -> t.Optional[datetime.datetime]:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	206 try:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	207 with path.open('r') as infile:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	208 unix_ts = float(infile.read())
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	209 except (OSError, ValueError):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	210 # If the last-written file is missing or corrupt, assume it is dead.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	211 return None
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	212 return datetime.datetime.utcfromtimestamp(unix_ts).replace(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	213 tzinfo=pytz.UTC)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	214
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	215
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	216 BSON_OPTIONS = bson.DEFAULT_CODEC_OPTIONS.with_options(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	217 tz_aware=True, tzinfo=pytz.UTC)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	218
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	219
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	220 def bson_encode(data: t.Dict[str, object]) -> bytes:
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	221 return bson.BSON.encode(data, codec_options=BSON_OPTIONS)

Mercurial > personal > weatherlog

annotate weatherlog/logger.py @ 14:c01f9929ae38