Show Idle (> d.) Chans

Results 1 ... 40 found in asciilifeform for 'from:billymg crawler'

billymg: alright, i appreciate the info, will look more into SQLite and maybe do a test of it in the crawler. in the meantime might see about just adding some reconnect logic to these programs
billymg: my setup is fairly small/simple. only two programs writing (logger and crawler) and two reading (their respective wwws)
billymg: asciilifeform: so potentially the crawler is at times tying up postgres such that it times out for the logger?
billymg: hmm, actually possibly the crawler still has its connection now
billymg: the damn thing keeps losing its postgres connection (same thing happens to my crawler too, and they both stop working at the same time until restarted)
billymg: caught a bug in my crawler's genesis though, where two of the sql queries use a different index name than the one that gets defined when initializing from bitdash_schema.sql. i'll post a regrind of the genesis soon but if anyone runs into it the fix is to change the two instances of 'ON CONFLICT ON CONSTRAINT unique_host DO UPDATE SET' to 'ON CONFLICT ON CONSTRAINT
billymg: << this method works, was able to get the crawler running. i installed all the python libs i needed by specifying exact versions, e.g. `pip install -Iv psycopg2==2.8.6`, and at least with my small list of required deps (flask, psycopg2, and requests) all were available
billymg: asciilifeform: i plan to write up a complete guide for this build, including the source files and tarballs where not the default, once this is all done (so far have working mp-wp, just need the crawler and logotron now)
billymg: cgra: ^ essentially that list, though asciilifeform's watchglass has a configurable knob for 'peershots', my crawler has that set to 5, not sure what alf's watchglass is set to
billymg: ah, yeah, sorry about that. on the roadmap
billymg: << the 36 number is any TRB node the crawler has encountered since it started running on my server. the homepage has a now, perhaps more useful, trb nodes active in last 48hrs
billymg: my crawler uses watchglass via an 'import watchglass' statement at the top of the file, but i'm only using a couple methods out of it
billymg: asciilifeform: yeah, i swear when i was running my crawler previously, a month or so ago, trb nodes always returned reasonable number of nodes (double or low triple digit counts)
billymg: punkman: fake as in not even the heathen crawlers count them as real or ever having existed
billymg had also planned to return to tuning the actual crawler/probe code after first getting the results visible on the www
billymg: just restarted the crawler with peershots=5, it finishes scanning all nodes in the network in about 20 minutes
billymg: signpost: interesting, the crawler results do seem to show that it's capped somewhere at about 2000 (i've never seen higher than 2001)
billymg: the crawler www is now browsable
billymg: the prb crawler has an api, maybe later at some point i could add in some automated cross referencing
billymg: << nice, looking at this node helped me identify a bug in my crawler
billymg: asciilifeform: the peer lists have been captured (the crawler is now stores probe history up to N probes, as set in conf). i'll dump the results somewhere permanent before the cap is reached
billymg: asciilifeform: it's consistent across all trb nodes that my crawler has picked up, and all in the last 1-2 hours (first time i've observed it since running this thing)
billymg: asciilifeform: is your node? my crawler is showing that sometime in the last hour or so it jumped from around ~40 connected peers (mostly good) to ~1200 peers (mostly fake/spam)
billymg: i'm also looking at as a potential library for rendering charts/graphs on the crawler www, in case anyone has experience with either, or has other recommendations
billymg: asciilifeform: i'm also getting close to making the crawler site live, at least a basic version so that others can take a look and provide feedback. at that point i think i'll need to upgrade from my rk to a bigger rig, especially since i also want to run whaack's block explorer on there
billymg: trinque: fwiw my working on the btc network crawler and new www to display obvious centralization of network is to attract more hands
billymg: asciilifeform: from there if you added some watchglass methods for getting blocks i could then incorporate those into the crawler (if that is what you meant by trying to analyze block propagation)
billymg: as soon as that's up will publish genesis for the crawler portion
billymg: my goal in making this crawler is to get more "bitcoiners" running trb nodes, and i suspect some al gore / nate silver stats and infograffix will widen the pool of those who see there is a problem
billymg: the ~8500 "actual" nodes number seems to be inline with what heathen trackers report as the total number of nodes on the network (~9k), so i suspect the crawler is nearly complete in mapping out reachable nodes
billymg: good morning, asciilifeform. my crawler seems to have hit a spam vein on the network, total unique IPs in the db exploded over the last two days to over 100k (these all come only from what a node returns in a 'getaddr' request). of those, when subsequently interrogated, only 8580 respond with a valid version message
billymg: i initially looked at it with the idea of repurposing for my crawler, barfed at 1001 dependencies pulled in, then remembered, "hey, watchglass does this"
billymg: asciilifeform: lotta hoops to jump through in setting it up:
billymg: the updated version of the crawler has been humming along nicely since last night, it's now up to ~4900 nodes discovered (heathen sites report over 9000)
billymg: << this wasn't even on my mp-wp todo list but since using postgres for the crawler it's now jumped near the top
billymg: from here i'm just going to proceed with making the crawler send the correct version message depending on whether it's trying to reach a trb or prb node. but yes, perhaps could write it so it tries once with '99999' then tries with '70001' and records result
billymg: asciilifeform: ah, interesting. now i'm wondering what the crawler could do to coax a node into sending a 'heathen command' in a reasonable amount of time
billymg: asciilifeform: i think whaack was working on a replacement block explorer. this thing i'm building is much simpler, just a network crawler
billymg: my reason for doing so was because both bitnodes and stopped tracking trb with their crawlers. i used to be able to check from time to time to see how many trb nodes are out there
billymg: asciilifeform: i wrote a simple btc network crawler that uses watchglass for node probing and dumps the results into a postgres db. it's been running since yesterday afternoon, here are some stats so far: