Show Idle (>14 d.) Chans


← 2022-12-24 | 2022-12-26 →
05:53 jonsykkel asciilifeform wat exactly is this unordred_set with o(1) insert/delete?
05:54 jonsykkel either way i think there is simpler way to do wihtout 1800 magic vector structures
05:54 jonsykkel where u just decide on max packet rate and determine static bufer sizes based on that
05:54 jonsykkel it requires u to use system clok insted of packet.stamp to retain o(1) insert
05:54 jonsykkel but that dosnt mater cuz theres a upper bound on the difference between those. simply means bufer will have to be sized for worst case
05:55 jonsykkel suchwise http://zzz.st/up/81UNY3cA/
~ 4 hours 34 minutes ~
10:29 phf "asciilifeform discovers cpp std"
10:31 phf also merry christmas
10:32 phf cookies and spiked eggnog hangovers are the worst, i don't know how santa does it
~ 15 minutes ~
10:47 asciilifeform phf: asciilifeform did hard time in cpp mines, long ago. but mostly fughot the horrors.
10:48 shinohai Merry Christmas all
10:49 asciilifeform http://logs.bitdash.io/pest/2022-12-25#1019254 << a cpp11ism
10:49 bitbot Logged on 2022-12-25 05:53:54 jonsykkel: asciilifeform wat exactly is this unordred_set with o(1) insert/delete?
10:50 asciilifeform http://logs.bitdash.io/pest/2022-12-25#1019255 << wrote the actual proggy with cheapo linked lists rather than expandable arrays
10:50 bitbot Logged on 2022-12-25 05:54:23 jonsykkel: either way i think there is simpler way to do wihtout 1800 magic vector structures
10:52 asciilifeform http://logs.bitdash.io/pest/2022-12-25#1019256 << this was the 1st algo asciilifeform considered, and rejected -- it often lets entries sit far longer than they gotta, and isn't much less expensive than asciilifeform's proposed algo afaik
10:52 bitbot Logged on 2022-12-25 05:54:26 jonsykkel: where u just decide on max packet rate and determine static bufer sizes based on that
10:52 asciilifeform ( likely -- substantially ~moar~ expensive, on acct of overripe entries )
10:55 asciilifeform prolly oughta test empirically tho
10:56 asciilifeform ( still not sure how jonsykkel's could possibly win anyffin, other than saving coupla kB of mem, in total, from having 1 linked list rather than 1800 )
10:59 asciilifeform per spec, ftr, 'A station’s Filter is a data structure which retains a hash of every valid incoming and originated Message, until such a time that if a duplicate of that Message were received, it would be deemed stale.'
11:00 asciilifeform wainot actually implement the spec.
11:03 asciilifeform observe that filter stores hashes of all msgs, not only chained; and emitted as well as received. letting'em all sit for potentially 15 extra min. is mega-bloat, esp. after we have signpost's luby
~ 1 hours 35 minutes ~
12:39 asciilifeform jonsykkel et al : runnable example of asciilifeform's filter . and somewhat braindead demo coad for same.
12:41 asciilifeform the 1 boojum asciilifeform can presently think of is that it aint threadsafe (needs lock, given as tick() runs async.)
12:45 asciilifeform ( obviously missing is the timer per se, and check for ticks <= maxTicks; but imho these belong in the packet receiver )
12:46 asciilifeform ditto re the check whether already contains $item.
12:48 * asciilifeform starts to think that in the process of posting this, reduced it to a trivial snoar. apologies to those who bored to tears.
12:49 * asciilifeform not intended 'verisimilitude flavour'
12:57 shinohai TP opus meum bunghole
12:57 shinohai >.>
12:57 asciilifeform lol
12:57 phf http://logs.nosuchlabs.com/log/pest/2022-12-25#1019348 << but on the other hand we don't want to discourage you writing experimental code, because unlike athena born as she was from the forehead of her father, distributed code has too much emergent behavior to attempt to reason it from first principles
12:57 dulapbot Logged on 2022-12-25 12:47:42 asciilifeform: starts to think that in the process of posting this, reduced it to a trivial snoar. apologies to those who bored to tears.
12:58 asciilifeform phf: per asciilifeform's pov, when writing field-expedient sadlang proggy, best one can do is to keep it small & correct, so to 1day rewrite
12:59 asciilifeform that being said, aint aboutta post erry cpp fart, except as to discuss algos
13:01 phf fwiw the code can be implemented as is in pest.lisp, because iit's intentionally not threaded and event based, including the 1s tick `periodic` funcion that does useful work
13:02 asciilifeform phf: asciilifeform's current attempt is forcibly threaded (on acct of the gui), and so decided that may as well properly bring out blackpacket eating to worker thrds, the redpacket processor -- 'consumer', etc
13:02 asciilifeform ( fwiw line-rate packet-eatin' reqs max parallelization )
13:06 * asciilifeform currently gnashing teeth re guiism idjicies -- e.g. on crapple, text entry box never gets default focus when displayed, no matter how baked, and nobody anywhere seems to know why
13:08 * asciilifeform has a rather 'half gui', where there's a widget that can display msgs in arbitrary order, and anuther 'tab' that displays console 'shell' through which 'peer x', 'key x key', etc. which actually control the proggy
13:16 phf 􏿽http://logs.nosuchlabs.com/log/pest/2022-12-25#1019359 << community consensus™ waivers on this point. i'm of twisted python school, and also did my time in realtime bidding in java saltmines where fixed ns to respond requirements were achieved exclusively with kevens on fds non-p
13:16 phf 􏿽arallel code, so i'm suspicious of max parallelization argument. though some modern™ approaches which have awareness of thread switching granularity and cache effects and modern™ architectures all rolled into one allegedly can outperform
13:17 dulapbot Logged on 2022-12-25 13:01:49 asciilifeform: ( fwiw line-rate packet-eatin' reqs max parallelization )
13:22 asciilifeform phf: feel free to check asciilifeform's napkin calculations re subj
13:22 bitbot (asciilifeform) 2021-09-23 asciilifeform: i expect you'd have at least 8cores on reasonable box, which gives you ~105840 ticks to process a packet, imho entirely doable.
13:23 asciilifeform seems clear that gb/s pestronics aint happening on 1 cpu core, no matter what kinda cpu.
13:24 asciilifeform eating blackpackets in parallel dun require anyffin 'modern' in re threading; is 'embarrasingly parallel'(tm)(r)(cray) sorta job
13:25 asciilifeform ( and also recall that the bigger the wot, the linearly moar work is to verify a blackpacket )
~ 42 minutes ~
14:08 jonsykkel http://logs.bitdash.io/pest/2022-12-25#1019265 << ah, "o(1) on average"
14:08 bitbot Logged on 2022-12-25 10:49:21 asciilifeform[6]: http://logs.bitdash.io/pest/2022-12-25#1019254 << a cpp11ism
14:08 jonsykkel after thikning about ur system with linked list and decent implementaiton, i agre my algo is bloat and trash
14:08 jonsykkel wastes ~2x memory
14:08 jonsykkel http://logs.bitdash.io/pest/2022-12-25#1019276 << luby frags will go in filter? this is imposible imo, if want to send fat warez at >dialup speed
14:08 bitbot Logged on 2022-12-25 11:03:13 asciilifeform[jonsykkel]: observe that filter stores hashes of all msgs, not only chained; and emitted as well as received. letting'em all sit for potentially 15 extra min. is mega-bloat, esp. after we have signpost's luby
14:10 jonsykkel there was some discusion re this http://logs.nosuchlabs.com/log/asciilifeform/2022-07-25#1112227
14:10 dulapbot (asciilifeform) 2022-07-25 asciilifeform: jonsykkel: there's no particular reason to dedupe warez fragments by looking in the db (or ever placing'em there to start with)
~ 25 minutes ~
14:35 asciilifeform jonsykkel: filter never touches disk (unless, i suppose, yer on a 20+yo machine where swapping is a thing )
14:36 asciilifeform jonsykkel: filter is absolutely req'd, per the dictum that no packet is to ever be processed 'as-new' >1nce; and in particular, in 0xfa draft, there are binary packets where no speaker field;
14:37 asciilifeform ... and the only way you can distinguish -- in the event a buggy station (or joker replaying captured packets) throws it back at you -- one you sent from one addrs'd to you -- is the filter
14:37 asciilifeform ( per new spec -- anyffin you send, chained or not, hits the filter before it hits the wire )
14:38 asciilifeform at the risk of pedanticism -- only ~chained~ msgs go to db on disk.
14:38 jonsykkel asciilifeform: yes, its in memory, where it will be imposibly big is my point, if its to store every luby frag
14:39 jonsykkel store hash of, that is
14:40 asciilifeform jonsykkel: you dun have to get it let 'impossibly' big (tho what do 15min of 32byte hashes of messages coming at 1g/s weigh? prolly coupla GB at most)
14:41 asciilifeform jonsykkel: ... instead can cap it, and start ignoring incomings until space frees
14:41 asciilifeform but having the deduper aint optional, if you accept 'not dupe' on faith at ANY point, yer open to replays
14:43 asciilifeform note that randos can't fill yer dedupe filter (it only stores hashes of validated, i.e. from actual peer, incl. self, and ~not previously seen~ msgs)
14:44 asciilifeform the most that can be done by rando replayer is to capture e.g. MB of legit packets and play it back to you until its timestamps expire. (which won't grow the filter, as they'll be dupes)
14:45 asciilifeform in other words, filter simply contains hashes of ~15min (avg) legit traffic in both dirs.
14:48 asciilifeform jonsykkel: at ~226860 pest packets/s ( see here ) , and 32min worth ,
14:48 bitbot (asciilifeform) 2021-09-23 asciilifeform: PeterL: let's assume ethernet. so, in bytes, let's calculate 1 packet's mass: 5 (gap) + 4 (preamble) + 14 (eth header) + 20 (ipv4 hdr) + 8 (udp header) + 496 (pest) + 4 (ethernet crc) == 551
14:49 asciilifeform ... and 32 byte per (hash), 32 * 60 * 30 * 226860 / (1024**2) == ~12462 MB max. (not incl. overhead of the set mechanism)
14:49 asciilifeform ( tho in practice you prolly won't be eating at 1g/s on yer box, the hmac thing will be bottleneck )
14:49 bitbot Logged on 2022-12-25 13:25:58 asciilifeform[5]: ( and also recall that the bigger the wot, the linearly moar work is to verify a blackpacket )
14:51 asciilifeform *30 min worth
14:52 asciilifeform if one were to narrow the time window to +/- 10min instead of 15 -- ~8.3GB, moar manageable
14:52 asciilifeform but in either case certainly not 'impossible' imho
14:55 jonsykkel the overhead of set mechanism as far as i can tell will double that number
14:58 jonsykkel then u gota traverse those (max) 200k long lists and delete 200k/sec from set (presumably hash table) as stuff starts expiring
14:59 jonsykkel maybe not "imposible" but significant, imo
15:01 jonsykkel conclusion from last linked thread was iirc no reason to put frags in filter cuz replays of these wont have any effect
~ 37 minutes ~
15:39 asciilifeform jonsykkel: depends on what sorta encoding used for the frags, neh
15:40 asciilifeform for 1 thing, there's no 'speaker' field in'em, so x->y can be passed back to x and he'll think is from y (if he hasn't got it in his filter) and vice-versa
15:41 asciilifeform for anuther thing, luby per se (even signpost's variant) doesn't automatically tolerate garbage
15:41 * asciilifeform not knows of 1 that does
15:42 asciilifeform naturally it tolerates missing/outta order frags. but not random piss (e.g. replayed from prev. sessions)
15:42 jonsykkel sure does, talking specifically about this scheme http://logs.nosuchlabs.com/log/asciilifeform/2022-07-25#1112236
15:42 dulapbot (asciilifeform) 2022-07-25 asciilifeform: each luby msg would include hash of the slice it is a frag of. thereby dr.evil piping in chunks of previous transfers would do 0
15:42 jonsykkel which does appear to tolerate random piss
15:43 asciilifeform jonsykkel: is how asciilifeform handled multipart chains, yes
15:43 asciilifeform jonsykkel: how about e.g. header packet tho, where one requests transmission of a file.
15:44 jonsykkel indeed would have to be restricted to the data fragments strictly
15:44 * asciilifeform holds that in general a station oughta always know 1) whether arbitrary msg is previously unseen 2) whether it had been sent by station itself or by a peer. 1+2 appears to require 'all goes in filter' in the general case.
15:46 jonsykkel agree that would be nice and simple, but at cost of 4 xtra ram sticks
15:46 asciilifeform if filter skips specifically unchained direct msgs of types which tolerate dupes cleanly, then imho acceptable
15:46 asciilifeform (ugly but acceptable, not errybody has 256GB of stix installed )
15:48 asciilifeform problem is that sumbody can eat yer cpu 100% by replaying luby frags, tho
15:49 asciilifeform key intent of pest is that a station can be made as ddos-proof as the external net pipe(s) allow.
15:50 asciilifeform if can guarantee that a replayed luby frag is rejected as fast (or faster) as the filter would've -- then, worx
15:50 jonsykkel posible, hard to say which is worse, jumping around randomly in memory 100k's per sec to delete from filter set is not great
15:51 * asciilifeform suspects that replayed lubyism could be rejected faster than a lookup in 16GB+ heap
15:52 asciilifeform re 'how' will prolly have to wait until the lubytron actually specified.
15:52 jonsykkel yep
15:59 asciilifeform for thread-completeness, anuther solution could be to actually track clock diff. b/w you and $peer, and treat binary msgs as expiring in e.g. 1min from timestamp (adjusted for the diff) rather than 15.
16:00 asciilifeform ( for arbitrary msg, +/-15min is forced by 'no guarantee that clocks are synced, gotta be able to speak if they aint' but for rapid-fire sequence of msgs, this aint a concern, the clocks won't drift on time scale of minutes or even hours by any substantial amt
16:00 asciilifeform )
16:02 asciilifeform you'd still be eating the cpu cost of working the filter heap, but in this case of over9000x smaller one.
16:03 asciilifeform ( if we suppose that under no reasonable circumstances will a packet >~1min. old in physical time need to be considered legit)
16:03 asciilifeform ... if someone sets up a pestron on mars, will need to revisit this, lol. but until then, imho entirely feasible to say 'msg is valid for 1min of physical time'
16:05 asciilifeform so, to run with this, suppose that 1) msg from cold peer (i.e. 1 that hasn't transmitted for $interval) is valid in +/-15m window 2) msg from 'warm' peer is valid for 1min (following drift-adjusted timestamp)
16:06 asciilifeform (1min may even be excessively generous. ever seen a 1min ping time?)
16:06 asciilifeform ^ jonsykkel awt phf signpost et al ^ plox to comment.
16:09 jonsykkel heap would be smaller, memory problem less bad, but same # of ops for cpu
16:10 asciilifeform jonsykkel: not same, as patently stale (per timestamp) msg only gets hash-compared to list of active getdata's, and not touched the filter
16:10 asciilifeform no need to look in the filter if can throw out as stale.
16:11 asciilifeform the +/-15min for ~all~ msgs, on reflection, is lunacy imho
16:12 asciilifeform it only makes sense if you've gone at least coupla hrs b/w last contact with $peer
16:13 jonsykkel right, not same for stales, but common case during lubytronic activities is that u are blasted with valid packets
16:14 * asciilifeform seen ping times in the range of coupla sec., when over ancient 'slip' etc., but nuffin remotely approaching 1min
16:15 jonsykkel maybe can be achieved with tcp tunnel
16:16 asciilifeform imho if you have ~1min pings and you aint operating a space fleet, you have serious problems that oughta be dealt with before trying to operate pestron
16:16 asciilifeform ( pestron is not intended for 'sneakernet', carrier pigeon, etc. systems )
16:17 asciilifeform if yer sneakernetting, send pgpgrams.
16:19 * asciilifeform must bbl
16:22 jonsykkel will have to shelve my plans of martian carrier pigeon experiments
~ 55 minutes ~
17:18 shinohai $ticker btc usd
17:18 busybot Current BTC price in USD: $16823.9
17:20 * shinohai views bot latency times .... "I'm ok with dat!"
17:32 * asciilifeform currently thinking: 1st packet from cold peer -- +/-15min; packet from warm peer valid +/- 15sec (after getting drift delta from the 'uncolding' packet). would need to think how behaves under replay scenario tho
17:33 asciilifeform i.e., theoretically, someone could replay an 'uncolding' old packet within 15min of its expir, and throw in the rest, and some of'em will be seen as fresh (given as they were evicted from filter)
17:34 asciilifeform needs moarthought imho.
~ 4 hours 13 minutes ~
21:47 phf 􏿽http://logs.nosuchlabs.com/log/pest/2022-12-25#1019365 << that's five hits of grateful dead grade acid level of architectural space man calculations. they look correct, and possible provide reasonable reference limits on things, but considering that you're posting sample code tha
21:48 phf 􏿽http://logs.nosuchlabs.com/log/pest/2022-12-25#1019365 << that's five hits of grateful dead grade acid level of architectural space man calculations. they look correct, and possible provide reasonable reference limits on things, but considering that you're posting sample code that
21:48 phf 􏿽uses c++ std, and i'm writing common lisp, we're not even in the same u.s. state as the building which has a room, where the practical conversation about this grade of solution takes place.
21:51 dulapbot Logged on 2022-12-25 13:21:39 asciilifeform: phf: feel free to check asciilifeform's napkin calculations re subj
21:51 bitbot (asciilifeform) 2021-09-23 asciilifeform: i expect you'd have at least 8cores on reasonable box, which gives you ~105840 ticks to process a packet, imho entirely doable.
21:52 dulapbot Logged on 2022-12-25 13:21:39 asciilifeform: phf: feel free to check asciilifeform's napkin calculations re subj
21:52 bitbot (asciilifeform) 2021-09-23 asciilifeform: i expect you'd have at least 8cores on reasonable box, which gives you ~105840 ticks to process a packet, imho entirely doable.
21:53 asciilifeform oh hm phf is that a duped msg or local station oops ?
21:54 phf and the other question is, where's the rest of it?
21:54 asciilifeform phf: seeing same item on desk as visible in billymg's logger
21:55 phf ah yeah i see it on billymg's logger also. not on nosuchlabs. maybe eventually?
21:56 asciilifeform phf: re the visible paragraph -- afaik 'max packets/sec can enter via a gb/s nic?' is hard bound; but the rest of thread is obv. dep. on internals of filter
21:56 bitbot (asciilifeform) 2021-09-23 asciilifeform: PeterL: let's assume ethernet. so, in bytes, let's calculate 1 packet's mass: 5 (gap) + 4 (preamble) + 14 (eth header) + 20 (ipv4 hdr) + 8 (udp header) + 496 (pest) + 4 (ethernet crc) == 551
21:57 asciilifeform (i.e. obv. nobody -- or certainly not asciilifeform -- knows how much the cppistic heap weighs when stuffed fulla x hashes, then y deleted, etc )
21:58 asciilifeform prolly depends on phase of moon, whether jupiter aligned with mars, gcc version, etc. rubbish.
21:58 asciilifeform the same, unfortunately, is true of ada's equiv. to this datastructure
21:59 asciilifeform 'hash maps' aint intended, apparently, for people who truly giveafuck re precise # of bytes occupies. (these -- write own structure, and not resort 'like lamer' to what comes with compiler)
21:59 phf `we wrote own equivalents of following c++ std libs to support necessary hard guarantees` etc.
21:59 asciilifeform aaha!
22:00 asciilifeform so phf 100% correct, no hard bounds avail. nor likely to be until thing is reimplemented 'ffa-style'
22:00 * asciilifeform skeptical even re bounds of a commonlisp variant, if there's any consing happening at all
22:02 * asciilifeform recalls in '15 when went on a lengthy chase re 'why trb hashmap grows unboundedly'
22:03 phf 􏿽common lisp solution is to refine the algo from naive and elegant to tight an heavily annotated, but mostly give up somewhere half way in once you discover that everyone still mostly uses the protocol to send text messages at 1message/minute to a group of 10 friends. end up outperf
22:04 phf 􏿽orming all competitors at this particular task.
22:05 asciilifeform asciilifeform's concern is when 1 or more of the 10 start shooting multi-GB luby pissstreams to 1 anuther
22:06 asciilifeform btw revisiting cppistic hashmaps -- they're retarded in moar ways than 1 (tho in some cases theoretically fixable). for instance, defaults to actually hashing (unsurprise) the inputs, even when, as in our case, is already a fixed-length hash
22:06 phf crashing the one canonical python implementation and bring the whole network down
22:06 asciilifeform ( theoretically overridable w/out resorting to compiler-proprietary gnarl. but not tried yet )
22:07 asciilifeform phf: atm even that 'send erryone a % command to run' lolbug still standing
22:10 phf 􏿽i'd probably put theoretical future luby code into a separate conditional branch, have totally special caching etc. strategy for the messages. at which point the problem of black->red as fast as possible becomes most important, and there's plenty of work even there. maybe ffi to so
22:11 phf 􏿽me c lib, maybe write an avx vectorized version, etc.
22:11 asciilifeform asmistic hmac384 would be an obvious win imho
22:11 asciilifeform ditto serpent
22:12 * asciilifeform has a half-written x64 serpent sitting around for yrs
22:13 asciilifeform ( aand even half of fpga serpent lol )
22:18 asciilifeform for thread-completeness, there's of course the obvious mega-cheat where one actually looks at the ip from which got the packet, and try the last-good key 1st, rather than rando-rotating through wot. but asciilifeform in the past explained wai 'is cheat', and 'forbidden in spec'
22:19 asciilifeform ( it blows away ability to compute worst-case performance, i.e. under flood of martians )
22:20 asciilifeform ( as well as making it easier for third party to distinguish station processing a martian from where processes a valid packet, via time sidechannel )
22:22 asciilifeform and ftr, the other thing -- pest is intended to be usable with radio, where no such thing as 'origin ip'.
22:22 asciilifeform ( i.e. errybody transmits to errybody )
22:22 asciilifeform this is atm 'bridge too far', but 'can't throw a word out of a song'(tm)
22:25 asciilifeform the radio not yet built, and who knows if/when. but wainot simulate the computational pain ~nao~, so not find a surprise later, 'we've the radio, but no one has fast enuff cpu to actually use it'
22:26 * asciilifeform not eager to preside over a repeat of his earlier moronic 'surprise' where 'no pc is fast enuff to 4096-bit rsa with arbitrary exponents at gb/s linerate'
← 2022-12-24 | 2022-12-26 →