personal/weatherlog: weatherlog/logger

annotate weatherlog/logger_test.py @ 14:c01f9929ae38

Make logger and HTTP writer more general and resilient. This makes the logger and HTTP writer more general, by removing any dependency upon the exact data type they are writing. They can now handle any type of BSON-serializable dict, and track what they have sent by keeping track of the last *byte*, not the last timestamp.

author	Paul Fisher <paul@pfish.zone>
date	Tue, 15 Oct 2019 22:40:24 -0400
parents	8a350ec1aa78
children	770215590d80

rev	line source
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	1 import contextlib
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	2 import datetime
6 8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	3 import fcntl
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	4 import itertools
6 8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	5 import multiprocessing
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	6 import pathlib
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	7 import tempfile
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	8 import time
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	9 import unittest
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	10
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	11 import bson
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	12 import pytz
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	13
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	14 from . import logger
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	15 from . import types
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	16
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	17
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	18 class FakeWriter(logger.RemoteWriter):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	19
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	20 BATCH_SIZE = 2
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	21
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	22 def __init__(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	23 self.writes = []
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	24
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	25 def write(self, readings):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	26 self.writes.append(tuple(readings))
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	27
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	28
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	29 class FlakyWriter(logger.RemoteWriter):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	30 is_bad = False
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	31
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	32 def __init__(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	33 self.writer = FakeWriter()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	34
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	35 def write(self, readings):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	36 if self.is_bad:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	37 raise logger.RemoteWriteError('I am bad!')
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	38 self.writer.write(readings)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	39
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	40 @property
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	41 def writes(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	42 return self.writer.writes
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	43
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	44
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	45 def ts(t):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	46 return datetime.datetime.utcfromtimestamp(t).replace(tzinfo=pytz.UTC)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	47
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	48
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	49 bs = logger.bson_encode
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	50
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	51
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	52 class LoggerTest(unittest.TestCase):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	53
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	54 maxDiff = None
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	55
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	56 def setUp(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	57 super().setUp()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	58 self.temp_dir = tempfile.TemporaryDirectory()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	59 self.temp_path = pathlib.Path(self.temp_dir.name)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	60
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	61 def tearDown(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	62 self.temp_dir.cleanup()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	63 super().tearDown()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	64
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	65 def test_from_nothing(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	66 writer = FakeWriter()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	67 with contextlib.closing(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	68 logger.BufferedLogger(self.temp_dir.name, writer)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	69 ) as bl:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	70 bl.WAIT_TIME = 0.2
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	71 bl.write({'first': 'item'})
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	72 bl.write({'entry': 2})
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	73 bl.write({'thing': 'three'})
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	74 bl.start()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	75 time.sleep(1) # Ensure that we get an entire logger cycle in.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	76
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	77 self.assertEqual(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	78 writer.writes,
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	79 [
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	80 (bs({'first': 'item'}), bs({'entry': 2})),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	81 (bs({'thing': 'three'}),),
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	82 ],
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	83 )
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	84 self.assertEqual(self._read_last_sent(), '59')
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	85 self.assertEqual(self._read_bsons(), [
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	86 {'first': 'item'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	87 {'entry': 2},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	88 {'thing': 'three'},
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	89 ])
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	90
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	91 def test_append_and_resume(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	92 existing_values = [
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	93 dict(sample_time=ts(10), temp_c=20, rh_pct=30),
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	94 dict(sample_time=ts(60), temp_c=10, rh_pct=99),
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	95 ]
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	96 with (self.temp_path / logger.BSON_FILENAME).open('wb') as outfile:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	97 for value in existing_values:
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	98 outfile.write(bs(value))
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	99 outfile.write(b'non-BSON garbage')
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	100
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	101 with (self.temp_path / logger.OLD_LAST_TS).open('w') as outfile:
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	102 outfile.write('10')
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	103
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	104 writer = FakeWriter()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	105 with contextlib.closing(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	106 logger.BufferedLogger(str(self.temp_path), writer)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	107 ) as bl:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	108 bl.WAIT_TIME = 0.2
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	109 bl.start()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	110 time.sleep(0.5)
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	111 bl.write({'some new': 'entry'})
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	112 time.sleep(0.5)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	113
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	114 self.assertEqual(self._read_last_sent(), '125')
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	115 self.assertEqual(self._read_bsons(), [
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	116 dict(sample_time=ts(10), temp_c=20, rh_pct=30),
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	117 dict(sample_time=ts(60), temp_c=10, rh_pct=99),
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	118 {'some new': 'entry'},
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	119 ])
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	120 self.assertEqual(list(itertools.chain.from_iterable(writer.writes)), [
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	121 bs(dict(sample_time=ts(60), temp_c=10, rh_pct=99)),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	122 bs({'some new': 'entry'}),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	123 ])
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	124
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	125 def test_resume_from_byte(self):
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	126 existing_values = [
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	127 {'old': 'value'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	128 {'unsent': 'value'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	129 ]
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	130 with (self.temp_path / logger.BSON_FILENAME).open('wb') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	131 for value in existing_values:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	132 outfile.write(bs(value))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	133 outfile.write(b'non-BSON garbage')
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	134 with (self.temp_path / logger.START_BYTE).open('w') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	135 outfile.write('20') # immediately after 'old: value'
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	136
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	137 writer = FakeWriter()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	138 with contextlib.closing(
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	139 logger.BufferedLogger(str(self.temp_path), writer)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	140 ) as bl:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	141 bl.WAIT_TIME = 0.2
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	142 bl.start()
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	143 bl.write({'some new': 'entry'})
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	144 time.sleep(0.5)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	145
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	146 self.assertEqual(self._read_last_sent(), '68')
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	147 self.assertEqual(self._read_bsons(), [
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	148 {'old': 'value'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	149 {'unsent': 'value'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	150 {'some new': 'entry'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	151 ])
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	152 self.assertEqual(list(itertools.chain.from_iterable(writer.writes)), [
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	153 bs({'unsent': 'value'}),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	154 bs({'some new': 'entry'}),
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	155 ])
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	156
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	157 def test_send_failure(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	158 writer = FlakyWriter()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	159 with contextlib.closing(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	160 logger.BufferedLogger(str(self.temp_path), writer)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	161 ) as bl:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	162 bl.WAIT_TIME = 0.2
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	163 bl.start()
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	164 bl.write({'cool write': 'succeeds'})
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	165 time.sleep(0.5)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	166 writer.is_bad = True
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	167 bl.write({'bad write': 'fails'})
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	168 time.sleep(0.5)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	169
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	170 self.assertEqual(self._read_last_sent(), '30')
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	171 self.assertEqual(
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	172 writer.writes, [(bs({'cool write': 'succeeds'}),)])
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	173 self.assertEqual(self._read_bsons(), [
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	174 {'cool write': 'succeeds'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	175 {'bad write': 'fails'},
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	176 ])
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	177
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	178 # Ensure that we resume writing again when the condition clears.
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	179
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	180 writer.is_bad = False
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	181 time.sleep(0.5)
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	182 self.assertEqual(self._read_last_sent(), '56')
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	183 self.assertEqual(
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	184 writer.writes,
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	185 [
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	186 (bs({'cool write': 'succeeds'}),),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	187 (bs({'bad write': 'fails'}),),
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	188 ])
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	189 self.assertEqual(self._read_bsons(), [
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	190 {'cool write': 'succeeds'},
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	191 {'bad write': 'fails'},
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	192 ])
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	193
6 8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	194 def test_fail_upon_lock(self):
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	195 bson_file = str(self.temp_path / logger.BSON_FILENAME)
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	196 out_queue = multiprocessing.Queue()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	197 in_queue = multiprocessing.Queue()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	198 # This needs to be in a separate multiprocessing.Process
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	199 # since flock-based file locks are per-process, not per-thread.
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	200 proc = multiprocessing.Process(
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	201 target=_lock_holder, args=(bson_file, out_queue, in_queue))
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	202 proc.start()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	203 in_queue.get() # Wait for the lock to be acquired.
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	204
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	205 with self.assertRaises(OSError):
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	206 logger.BufferedLogger(str(self.temp_path), FakeWriter())
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	207 out_queue.put(None) # Notify that we're done.
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	208 out_queue.close()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	209 proc.join()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	210 proc.close()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	211
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	212 # Test that it works after the lock is released.
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	213 logger.BufferedLogger(str(self.temp_path), FakeWriter()).close()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	214
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	215 def test_upgrade_last_sent(self):
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	216 for (timestamp, byte_count) in [
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	217 ('5', '0'),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	218 ('20', '52'),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	219 ('30', '78'),
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	220 ]:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	221 bson_file = self.temp_path / logger.BSON_FILENAME
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	222 with bson_file.open('wb') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	223 outfile.write(logger.bson_encode(dict(sample_time=ts(10))))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	224 outfile.write(logger.bson_encode(dict(sample_time=ts(20))))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	225 outfile.write(logger.bson_encode(dict(sample_time=ts(30))))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	226 outfile.write(b'some bogus data')
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	227 last_sent = self.temp_path / logger.OLD_LAST_TS
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	228 with last_sent.open('w') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	229 outfile.write(timestamp)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	230 start_byte = self.temp_path / logger.START_BYTE
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	231 with bson_file.open('r+b') as infile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	232 logger._read_unsent_and_upgrade(
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	233 infile, last_sent, start_byte)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	234 self.assertFalse(last_sent.exists())
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	235 with start_byte.open('r') as infile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	236 self.assertEqual(infile.read(), byte_count)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	237
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	238 def test_upgrade_last_sent_no_last_sent(self):
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	239 bson_file = self.temp_path / logger.BSON_FILENAME
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	240 with bson_file.open('wb') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	241 outfile.write(logger.bson_encode(dict(sample_time=ts(10))))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	242 outfile.write(logger.bson_encode(dict(sample_time=ts(20))))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	243 outfile.write(logger.bson_encode(dict(sample_time=ts(30))))
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	244 last_sent = self.temp_path / logger.OLD_LAST_TS
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	245 start_byte = self.temp_path / logger.START_BYTE
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	246 with start_byte.open('w') as outfile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	247 outfile.write('untouched')
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	248 with bson_file.open('r+b') as infile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	249 logger._read_unsent_and_upgrade(
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	250 infile, last_sent, start_byte)
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	251 self.assertFalse(last_sent.exists())
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	252 with start_byte.open('r') as infile:
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	253 self.assertEqual(infile.read(), 'untouched')
c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	254
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	255 def _read_last_sent(self):
14 c01f9929ae38 Make logger and HTTP writer more general and resilient. Paul Fisher <paul@pfish.zone> parents: 6 diff changeset	256 with (self.temp_path / logger.START_BYTE).open('r') as infile:
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	257 return infile.read()
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	258
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	259 def _read_bsons(self):
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	260 with (self.temp_path / logger.BSON_FILENAME).open('rb') as infile:
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	261 return bson.decode_all(infile.read(), logger.BSON_OPTIONS)
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	262
885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	263
6 8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	264 def _lock_holder(path, in_queue, out_queue):
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	265 with open(path, 'w') as infile:
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	266 fcntl.flock(infile, fcntl.LOCK_SH)
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	267 out_queue.put(None) # Notify that we've acquired the lock.
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	268 out_queue.close()
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	269 in_queue.get() # Wait for the test to complete before closing.
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	270
8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	271
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	272 if __name__ == '__main__':
6 8a350ec1aa78 logger_test: Ensure it crashes when the file is locked. Paul Fisher <paul@pfish.zone> parents: 5 diff changeset	273 multiprocessing.set_start_method('spawn')
5 885bff085edf Add remote logger class for eventual HTTP writer. Paul Fisher <paul@pfish.zone> parents: diff changeset	274 unittest.main()

Mercurial > personal > weatherlog

annotate weatherlog/logger_test.py @ 14:c01f9929ae38