#pest | 2022-06-28

$ticker btc usd

14:46

busybot

Current BTC price in USD: $20760.79

14:47

!c net-summary

14:47

Bitcoin Network (IPv4 Nodes Active Within the Last 48 hours) Global: 7623; TRB-Compatible: 31; TRB: 13

14:47

TRB-Compatible by Country: United States: 7; Germany: 4; Romania: 3; Singapore: 3; Russia: 3; France: 2; Spain: 2; Canada: 1; Ukraine: 1; Italy: 1; Algeria: 1; Sweden: 1; Belgium: 1; Bulgaria: 1;

14:47

TRB by Country: United States: 7; Canada: 1; Romania: 1; Lithuania: 1; France: 1; New Zealand: 1; Norway: 1;

14:47

!c trb-status

14:47

75.106.222.93 (Alive), h=742714, v=99999, United States - peers: 130 - last probed: 38m ago

14:47

71.114.46.117 (Alive), h=742715, v=99999, United States - peers: 90 - last probed: 35m ago

14:48

54.38.94.63 (Alive), h=742714, v=88888, France - peers: 68 - last probed: 40m ago

14:48

82.79.58.192 (Alive), h=742714, v=99999, Romania - peers: 64 - last probed: 39m ago

14:48

85.164.243.42 (Alive), h=742715, v=99999, Norway - peers: 62 - last probed: 34m ago

14:48

94.176.238.102 (Alive), h=742357, v=99999, Lithuania - peers: 59 - last probed: 39m ago

14:48

205.134.172.4 (Alive), h=742714, v=70001, United States - peers: 56 - last probed: 39m ago

14:48

54.39.156.171 (Alive), h=742714, v=99999, Canada - peers: 45 - last probed: 39m ago

14:48

205.134.172.28 (Alive), h=742714, v=99999, United States - peers: 39 - last probed: 40m ago

14:49

205.134.172.26 (Alive), h=742714, v=99999, United States - peers: 26 - last probed: 39m ago

14:49

shinohai

oh hey a Norwegian trb node

14:51

103.6.212.28 (Could not connect!), h=742599, v=99999, New Zealand - peers: 15 - last probed: 37m ago

14:51

i think it's jonsykkel's

14:51

205.134.172.27 (Alive), h=742714, v=99999, United States - peers: 1 - last probed: 39m ago

14:51

205.134.172.6 (Alive), h=742714, v=99999, United States - peers: 14 - last probed: 40m ago

~ 58 minutes ~

15:49

While debugging I noticed that one thing that might be slowing down the initial sync is that almost every GETDATA response necessarily triggers a db query.

15:51

Most of the slowness is likely due to blatta being single threaded, however. For ex., I can see message handling visibly pause while output is being dumped to the IRC client.

16:01

yeah, I bumped into python's threading limitations with ocpy too (that onlinecodes encoder)

16:04

you might be able to make a clean separation of the pest protocol handling and IRC handling using multiprocessing.

16:04

huck messages back and forth between them over Queues or something.

16:08

Another win might be to separate all db writing/reading ops into another process

16:09

asciilifeform

signpost: imho ideally ircism in entirely separate proggy eventually

16:09

dulapbot

(asciilifeform) 2022-06-28 asciilifeform: the irc frontend is very much, imho, a straightjacket

16:12

multiprocessing's pretty much that, fires up separate processes and gives ya a python-shaped hole to yap between them.

16:12

but yeah, w/e makes the irc part easiest to cleave off.

~ 27 minutes ~

16:39

Ok so best post mortem I can come up with right now for the dupe GETDATA requests is: occasionally stations with very old netchain send a message. When doing initial sync, these are were encountered faster than the order buffer was getting dumped.

16:40

PROD will fix this. Also, increasing the length of time outgoing GETDATA requests are retained in memory helps.

16:43

Also planning to add a knob to prevent dumping messages older than x into the IRC client.

16:51

As already discussed, in the future, loggers should read directly from the station log. I may write a console tool to dump the logs in order in a readable format to verify syncs are completing successfully.

~ 43 minutes ~

17:35

imho I wouldn't even bother avoiding dumping old messages to the IRC console at this point.

17:36

that'd be another example of letting accidental complexity creep in from the temporary measure of using IRC for UI.

17:36

later "oops, special case that avoided sending this message to IRC also avoided sending it to db"

17:37

giving that as an example of the kinds of bugs this backflow of temporary measures into core design can cause.

17:37

I don't think anybody's terribly inconvenienced atm, and I think the line oughta be held on hacks.

17:38

log-reader oughta read the db, yes, not the IRC adapter. db is the source of truth on message chain state, not the UI.

17:38