Show Idle (>14 d.) Chans


← 2020-05-14 | 2020-05-16 →
13:58 jfw asciilifeform: mind expanding on what you mean by leave it the fuck alone and why?
13:58 snsabot Logged on 2020-05-11 13:23:37 asciilifeform: asciilifeform's pov re trb worx ftr
13:59 jfw you think it's good enough? readable? works well? "close enough for government work" as the contractors said back home?
14:00 asciilifeform jfw: i'll elaborate (in ~15 m, tied up in meatspace atm)
14:00 jfw no rush.
14:03 asciilifeform jfw: are you familiar with how i stepped into that pile o'shit to begin with, in '14 ?
14:05 asciilifeform my notion was 'we'll rewrite the client'. but 1st thought 'let's find what original does', as it was never 100% documented anywhere. mp contributed a copy of 0.5.3. asciilifeform then cut quarter MB of obvious rubbish from it, and signed (this was pre-'v'.) various other folx started reading, cutting.
14:05 jfw not entirely, I know you were btc-skeptical at first, and MP invited you to review his exchange
14:06 asciilifeform asciilifeform's orig. notion was to produce 'the book', i.e. something like lion's unix 6th ed. for bitcoin .
14:06 asciilifeform jfw: this was already 1+y after asciilifeform was persuaded to join mp et al.
14:07 asciilifeform the orig. threads on tbf ml.
14:07 jfw right, there was a 'gypsy chicken recipe' involved back then
14:07 asciilifeform right, that was where i took the 0.5.3 and cut off the mswin, gui, buncha cruft
14:08 asciilifeform phf's patch viewer still best imho illustration of what/when cut, by whom
14:09 jfw I've read or at least skimmed all the vpatches ftr, signed & hosted at http://fixpoint.welshcomputing.com/v/bitcoin/
14:09 asciilifeform for long time, project was stalled -- 0.5.3 would sync up to certain block and mysteriously grind to halt. then in fact mp dug up the pill, was bug in the orig. bdb. then became possible to run working nodes .
14:10 jfw insufficient magic numbers in the bdb locking system config
14:10 asciilifeform aha
14:11 asciilifeform so, to return to jfw's orig. q: to date, ~errything that's been done to trb, consisted of a) cuts of dross (the irc seed finder, mswin liquishit, gui, etc) b) bug fixes (most recently : the 'getinfo' wedge)
14:11 jfw (to be pedantic, 'at least skimmed' does not apply to genesis, though I've started on a gradual full readthrough.)
14:11 asciilifeform the thing is defo not pleasant to read, or in any way sane example of programming , by even microshit standards
14:12 asciilifeform asciilifeform cannot speak for all of the participants, but what he wanted it for, was as ~reference~
14:12 asciilifeform i.e. so can write new client that's guaranteed to be 100% compat (i.e. won't spontaneously end up forked off at any pt)
14:13 asciilifeform any major change to the orig. item, potentially destroys this function (i.e. potentially changes the semantics of the thing in the face of new inputs) . mp called this the 'grandfather's pistols' problem .
14:14 asciilifeform so no one made any substantial change. ( asciilifeform at one time glued on a scheme interpreter, and 'let's rewrite pieces in that', but this didn't catch on, and abandoned. arguably it was ill-conceived idea from start. )
14:14 jfw yes, that's a challenge; otoh, I find it ~impossible to reason about what wtf those pistols currently do anyway
14:15 asciilifeform to give very concrete example : the recent 'wedge' was on acct of the nonsensical ocean of semaphore locks in trb. and would seem 'why not remove'em' . but go and try.
14:15 asciilifeform there's ~no way to guarantee exactly same behaviour as the orig. cuz orig author simply sprinkled locks around like a child.
14:17 jfw perhaps to make matters worse, I'm not sure there's any guarantee that original has same behavior as itself on different occasions.
14:17 asciilifeform exactly; the 'getinfo' wedge was specific instance of just this.
14:18 asciilifeform trb is literally the shoddiest piece of ??? asciilifeform ever had the misfortune to work with. and asciilifeform maintains legacy winblows crud for a living.
14:18 jfw the bdb locks too; I imagine it'd depend on how exactly your b-trees ended up layed out due to past history as to how many pages it needed to lock and thus what exact block it would stick on.
14:18 asciilifeform exactly. and think, wtf does the thing need a general-purpose db, when ~old blocks don't change~ ~
14:19 asciilifeform block >20 or so deep, will never change ! till sun burns out. why need b-trees ?!
14:19 jfw well for starters it needs to update an index of where txns are spent, no?
14:19 asciilifeform answer is, orig author was lazy ignoramus. and, on top of this, never finished the item.
14:20 asciilifeform jfw: this doesn't require a changeable db, w/ trees. i outlined a simple o(1) scheme some yrs ago, where one actually stores backlinks to the tx being spent.
14:21 jfw does that require changing the block format though?
14:21 asciilifeform nope
14:21 asciilifeform entirely client-size (i.e. follows traditional protocol when speaking)
14:22 asciilifeform jfw: posted some of the example coad a while ago .
14:23 asciilifeform ( see 'type Tx...' )
14:24 asciilifeform idea is, when you eat a block, you store the tx in such a way that they actually point to the ones being spent. in mmap'd array. the respective scripts are kept in another, so that they don't fuck your page alignments. (ea. tx points back to the orig. script. so can reconstitute orig. tx / block on demand )
14:25 jfw how do you find where the ones being spent are, if you don't have an index by txid?
14:25 asciilifeform in yet a 3rd mmap'd array you keep hash table. so you can take e.g. 1st 32bits of txid, and deref array, and in that place either a 0, or the index of where the tx lives in the tx array.
14:26 asciilifeform you do index by txid. and with ea. tx, keep the indices of any it spends, and any that in turn later spent it.
14:27 asciilifeform this would be very difficult if you had to handle reorgs. asciilifeform's discovery was that you don't have to, if you keep e.g. last 100blox in 'nursery' (something similar to the old db) and only >100-deep blox, in the static arrays.
14:27 asciilifeform then you don't need to support reorg, for the latter.
14:27 jfw but tx index is a tree then, no? (or hash table which isn't guaranteed access time)
14:28 asciilifeform guaranteed o(1) access.
14:28 asciilifeform i'll elaborate if it aint clear
14:28 asciilifeform say you want to look up a tx by id.
14:28 asciilifeform you take 1st 32bits of the id.
14:29 asciilifeform then you look at what's in index(first_32_of_id). it'll be a e.g. 48bit word. if it's 0, there's no tx with that id.
14:30 asciilifeform if it aint 0, that's an index into the blocks array .
14:30 asciilifeform both mmaped.
14:30 asciilifeform this requires a large, but, interestingly, roughly same mass of disk as the old db.
14:30 jfw and in case of collision of first 32 bits?
14:30 asciilifeform and because the index is a sparse array, mmap Does Right Thing, so it doesn't eat up all yer ram
14:31 asciilifeform jfw: in case of collision, the 1st of those 48 is set, and the pointed address contains the full list of all id's having those given 32bits in start.
14:31 asciilifeform my other suggestion , for minimizing collisions, was to pre-hash the txids with a machine-local salt, to keep people from 'mining' colliding txid's and grinding erryone down
14:32 asciilifeform but in fact you do have to handle collisions, elementarily, given as txid is quite bit longer than 32b
14:32 asciilifeform ftr i thought all of this was clear when said 'hash table...'
14:32 jfw sounds the same as any other hash table, so not sure why the objection to btree, but ok, maybe implementation ends up simpler.
14:32 asciilifeform cuz it doesn't need to change!
14:33 asciilifeform there's no reason to have to balance a tree for this.
14:33 asciilifeform simply suffices to lay down the tx as the come in, in the next available free space. and update the index appropriately.
14:33 asciilifeform (likewise gotta track which block # given tx belongs to. but this is imho obv. detail.)
14:35 jfw anyway, mind returning upstack for now? I'll assume this scheme works fine; where does that leave TRB work?
14:36 asciilifeform jfw: after described this, went to write a mmap lib for ada. this was where stopped.
14:36 snsabot (trilema) 2019-01-22 a111: Logged on 2017-06-06 19:40 asciilifeform: mod6, phf , et al : http://nosuchlabs.com/pub/ada/horsecocks.tar.gz << i dun recall posting this before, so here it will live, for nao : unofficial release of mmaptron
14:36 asciilifeform aite
14:36 asciilifeform jfw: imho trb is a 'life support' item. i.e. one needs a working client, not only to study 'let's rewrite', but to actually use bitcoin .
14:37 asciilifeform most of what i've done to the thing since '15, was from that pov. i.e. 'people report it takes 8 weeks to sync, let's see why'
14:37 asciilifeform 'nodes seem to all be wedged, let's find why.' etc
14:38 asciilifeform taking, for instance, sync (iirc jfw once said he was interested in why sync slow)
14:38 asciilifeform in '18 i profiled the thing, for months. found that spends ~100% of time ~waiting for someone to give new block~
14:38 asciilifeform cuz orig author was a moron, and made sync something that happens only briefly ~at boot time~
14:39 jfw re sync, quite; for instance I presently can't transact because I've had a node sitting on desk for >month that spending most of its time doing nothing but receiving bastard blocks until killed & restarted.
14:39 asciilifeform wrote 'aggression' patch. which seemed to help (changed logic to 'ask for blocks' at time new peer connects, rather than the idiot bootup )
14:40 asciilifeform but this also not cure. because if entire cluster of nodes not gets new blocks for while, but connections still alive, they happily sit there like idiots and get 0 new blocks.
14:40 asciilifeform ben_vulpes attempted a more aggressive sync, but iirc wasn't quite enuff .
14:41 asciilifeform afaik no one has since experimented with this bit ( it only comes into play when node falls severely behind, is possibly why )
14:41 asciilifeform jfw: 'bastard blocks' are simply blocks for which the antecedent is not known. it'll be 100% of incoming blocks , from a node's pov, if it lacks even one antecedent for'em.
14:42 asciilifeform i introduced the term in this patch .
14:42 jfw right.
14:42 asciilifeform ( orig. 0.5.3 had a data structure called 'orphanage' , which stored ~arbitrary~ incoming turds claiming to be blocks, and ate ~100% of memory )
14:42 jfw and I've nfi why trb peers send me non-current blocks for which I lack antecedents
14:43 asciilifeform cuz they don't know that you lack the antecedent .
14:43 jfw they just guess at what block numbers I need?
14:43 asciilifeform default behaviour is, for instance, to retransmit ~all~ incoming new blocks that got accepted by node.
14:43 asciilifeform they don't even guess. simply blindly relay.
14:44 asciilifeform and, moreover, even a node that is syncing, will relay. i'ma work an example :
14:44 jfw perhaps it's that - they're relaying new blocks only, and problem is that the sync 'ran out of steam' after leaving 'bootup' phase.
14:45 asciilifeform you have a new node, it's at 0. mr. x has another young node, it's at 100k. he just got 100,000 .. 100,500. his node peers with yours. yer gonna receive those 100,000 .. 100,500 , whether you want'em or not
14:45 asciilifeform whether you know what to do with'em, or not
14:45 asciilifeform cuz mr. x's node doesn't ~know~ the 'current block height' of whole net. it's an abstraction. it only knows that it just learned 500 'new' blox.
14:45 asciilifeform so 'helpfully' sends'em to you.
14:46 jfw sure, s/new/new-to-them/ then
14:46 asciilifeform i oversimplify slightly, the orig client has a hardcoded turd that tries to guess if it's 'severely behind' based on the block heights reported by peers. but it doesn't, as you might expect, work reliably to prevent this kinda spam.
14:46 asciilifeform the nodes default to sending 'more than you want', cuz that's how the blocks/tx propagate. it's primarily a 'push' mechanism.
14:48 jfw Now - perhaps this is where I interrupted re mmap db - is it your view that efforts at this point are best spent on rewrite while problems with the ref. impl. are handled reactively?
14:49 asciilifeform 1s
15:03 asciilifeform jfw: correct
15:07 jfw asciilifeform: alright, thanks.
15:08 jfw jurov, trinque: I'm curious whether you concur with the above. Gotta run now but will check log.
15:14 asciilifeform jfw: i'm not somehow opposed to efforts to civilize trb 'in sections'. but imho suffers from same problem as ab initio rewrite (i.e. not guaranteed to preserve semantics) while still retains the idjit 'serializations' etc
15:14 asciilifeform see also .
15:14 snsabot (asciilifeform) 2020-05-03 asciilifeform: the ~whole calamity of trb is that the internal representations for ~everything are thoroughly braindamaged. if you fix this, you get ~0 of the orig. trb remaining, i.e. same as total rewrite. if you don't, you get 100k loc of glue.
15:15 asciilifeform for my part, i'ma rip out the ssl crapola an' replace w/ ffa. (dunno if anyone ~else~ wants such a trb, but i'm personally tired of sslism)
15:16 asciilifeform this strictly from pov of ' asciilifeform's personal nodes ' , but of course will publish .
15:26 asciilifeform http://logs.nosuchlabs.com/log/therealbitcoin/2020-05-15#1000138 << btw i notice the wedge pill aint in there. may be 1 reason why jfw's box can't sync .
15:26 snsabot Logged on 2020-05-15 14:09:11 jfw: I've read or at least skimmed all the vpatches ftr, signed & hosted at http://fixpoint.welshcomputing.com/v/bitcoin/
~ 1 hours 18 minutes ~
16:44 trinque jfw: mmmm it's hard to say; the thing's a hellish tarpit for sure.
16:45 trinque were I taking it on, I'd probably pare the thing down a bit yet before rewriting
16:45 trinque I have a patch on my desk that cleaves off the entire wallet and stuff only used by wallet
16:45 trinque you're welcome to it if it's helpful
16:46 asciilifeform trinque: iirc jfw made something similar , prior to tuning in
16:46 trinque ah, cool then
16:46 trinque it does cut the weight a bit
16:46 trinque (not nearly enough but)
16:46 asciilifeform reduces the thickness of the 3ring you need to keep the thing in, for sure
16:47 asciilifeform but does 0 against 'serializations', the locking horror, etc
16:47 asciilifeform they're pretty firmly 'glued with broken glass'
16:47 trinque probably wireshark protocol analysis is a better path forward eh?
16:47 asciilifeform trinque: oddly enuff, the protocol i found to be quite simple. e.g. 'watchglass' in fact nao speaks 100% of it
16:47 trinque mhm
16:48 asciilifeform ( and indeed i had to wireshark , it aint 100% documented afaik anywhere )
16:48 trinque honestly if the piece of shit relies on implicit behavior, better out with it now than later
16:48 asciilifeform for double lulz, 'wireshark' comes w/ what claims to be bitcoin protocol decoder . it only quasi-worx.
16:49 trinque would be neat to see an item produced that horks and sprays blocks, and doesn't worry about whether it's doing it in exactly the same way as trb.
16:50 asciilifeform for instance, 1 find.
16:50 snsabot (asciilifeform) 2020-03-01 asciilifeform: shinohai: this btw confirmed. i did experiment, if one sends that extra byte, prb noades do in fact send addrs. (and yes some of'em ipv6, have to be thrown out)
16:50 asciilifeform ( re protocol corners. )
16:51 asciilifeform trinque: indeed would. and one could jettison some of the weird special cases (e.g. support for replaceable tx's) in the orig., if you cement in hashes for 1st 600k blox etc
16:55 asciilifeform ( iirc i described this in the past. cement, not a la prb's 'dun need to verify these', but rather 'oh and btw the 1st N blox have THESE hashes, and it someone brings in a 599,999 that doesn't, he's trying to fuck you )
16:56 asciilifeform the demo for this, on the ml.
~ 15 minutes ~
17:12 * asciilifeform will add : imho the main reason why trb seems to consist 80+% by mass of kilometres of pointless abstractions standing in each others' (and reader's) way -- is that this style is forced on cpp victims by the language; it is how they avoid coarse errors of pilotage (buffer off-by-1s and other consequences of c/cpp's idiot lack of ranged types, bounds checks, built-in array moves )
17:13 asciilifeform microshit's coad looks ~entirely like trb. (but -- a little cleaner, they have 1000 monkeys on erry piece of shit, sweeping up )
17:14 asciilifeform this is how 'oop'ism became a thing.
17:14 asciilifeform 'oh noez, if you expose the raw bytes, someone could off-by-1...'
17:20 asciilifeform for illustration, compare this to that .
17:23 asciilifeform ( for the innocent: the item on left , and the 1 on right, do nominally 'same thing' ... )
17:25 * asciilifeform would link to a cpp example, from heathendom, but folx will lose their lunch, so won't
~ 31 minutes ~
17:56 shinohai asciilifeform: Do you, offhand, know the link to the thread re: bdb bug ?
~ 30 minutes ~
18:27 asciilifeform shinohai: it's in phf's logs , defo not in mine, being 'era 1' (pre-apr of '16)
18:28 asciilifeform there .
18:29 shinohai oh fuck NOW I remember that. ty asciilifeform
18:30 asciilifeform np.
~ 18 minutes ~
18:49 jurov asciilifeform: i don't have node atm, the hdd died and waits for reinstall. no idea whose is the radiolan.sk one
18:49 jurov will ingest the rest of log tmrw
18:51 asciilifeform ah hm jurov i always assumed it was yours
18:51 * asciilifeform wonders who the seekrit lurker slovak were...
← 2020-05-14 | 2020-05-16 →