[cryptome] Re: full year 2014 corpus - better concurrent continue

  • From: coderman <coderman@xxxxxxxxx>
  • To: cryptome@xxxxxxxxxxxxx, cypherpunks@xxxxxxxxxx
  • Date: Sun, 11 Jan 2015 16:32:33 -0800

"why this format?"
 - it is lowest common denominator, in a sense. the rest explained later...

but first, a fix. there is an issue with commands that won't continue
past the PDF and small file bundles.  use fix below to run or more
continuation downloaders:

### The gist of a quick continue download: (many can be run in parallel)
# Update torsunget.sh and run dl-fy2014.sh to complete.
# assumes a copy of https://peertech.org/files/fy2014lst.txt
# or 84fe6b33e5b6f2478523432514fcb24b844105c4a38635ba97543e9c7152f90e
# in current working directory where ./dl-fy2014.sh is run.
# E.g.: curl -x -o fy2014lst.txt \
#  http://bigsun36arflx75h.onion/shid/84f/e6b/84fe6b33e5b6f247..7543e9c7152f90e
rm -f torsunget.sh >/dev/null 2>&1
curl -x -o torsunget.sh \
 http://bigsundaawafn36e.onion/torsunget.sh.txt \
  && chmod +x torsunget.sh
curl -x -o dl-fy2014.sh \
 http://bigsundaawafn36e.onion/dl-fy2014.sh.txt \
  && chmod +x dl-fy2014.sh
time ./dl-fy2014.sh

Other related posts:

  • » [cryptome] Re: full year 2014 corpus - better concurrent continue - coderman