jyapayne/worm-scraper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
314 commits 1 branch 27 tags 2 MiB
Commit graph

37 commits

Author SHA1 Message Date
Evan Young
12394da334
 		Fix failure on convert in modern Node.js 
Closes #30.
 2022-01-22 13:39:12 -05:00
Domenic Denicola
360f108c1c Use @domenic/eslint-config 2021-02-15 22:05:38 -05:00
Domenic Denicola
d5f3b58f0b Slightly improve program output 
Most notably reduce the number of output lines per chapter in the download step.
 2020-12-26 17:54:19 -05:00
Domenic Denicola
0be098ff16 Allow running with no commands 2020-12-26 17:43:42 -05:00
Domenic Denicola
58b0856deb Stop failing the convert step due to busy filesystems 
This often happens with virus scanners, etc.
 2020-11-14 20:26:44 -05:00
Domenic Denicola
4fbaa0e8db Settle on "Patrol <lowercase>" and "the Patrol" 
This is reversing the decision made in 84e0d83bd1, and extending the revised decision to other Patrol-related terms.
 2020-10-26 21:58:41 -04:00
Domenic Denicola
ad2c0b0ca5 Improve conversion speed using a worker pool 2020-10-26 21:58:40 -04:00
Domenic Denicola
354fb46819 Add per-book covers 2020-07-01 17:23:05 -04:00
Domenic Denicola
fd914b43f5 Start generalizing to work for Ward too 
* Updates the command-line options in various ways, notably changing --start-url to --book
* Renames the "book" directory to "staging"
* Abstracts out most book-specific data into books.js (but not yet cover or substitutions.json)
* Makes per-book subfolders under cache and staging directories
 2020-07-01 15:17:52 -04:00
Domenic Denicola
ecdadf8fd9 Update dependencies 
This includes eliminating some dependencies that are included in modern Node.js versions.
 2020-07-01 14:36:14 -04:00
Domenic Denicola
4386d48021 Add --jobs option for the convert command 
This doesn't seem that helpful as there seems to be a lot of sequential work going on regardless.
 2019-08-16 23:37:00 -04:00
Domenic Denicola
0435c45b2e Update dependencies and code style 2017-08-13 16:48:19 -04:00
Tejas Gupta
1af212447c Add shebang line 
This makes it work on non-Windows OSes. Fixes #5; fixes #4.
 2017-08-13 15:39:45 -04:00
Domenic Denicola
d5157f2c22 Update dev dependencies and use new JS features 2017-01-02 20:13:40 -05:00
Domenic Denicola
853b76a881 Remove unhandled rejection tracking 
Modern Node.js does this now.
 2017-01-02 19:34:55 -05:00
Domenic Denicola
876be19c8b Update dependencies 2017-01-02 19:27:48 -05:00
Domenic Denicola
48ddecfed4 Implement zip step 2015-05-21 02:17:44 -04:00
Domenic Denicola
23e79ed6ba Move copying of scaffolding files into scaffold.js 2015-05-21 02:17:43 -04:00
Domenic Denicola
59b82e7ab4 Make a nicer command-line interface 2015-05-21 02:17:43 -04:00
Domenic Denicola
247f713e13 Better cleanup in the convert step 2015-05-11 22:52:14 -04:00
Domenic Denicola
ace6a55be8 Update extras code to use cache manifest 
Finally we have reasonable chapter titles in the TOC.
 2015-05-11 21:24:16 -04:00
Domenic Denicola
cf6b5c9ab9 Update conversion to work with cache manifest 2015-05-11 21:24:16 -04:00
Domenic Denicola
b68b88e17e While downloading, save a cache manifest alongside 
This allows us to keep track of the chapter title after the fact.
 2015-05-11 21:24:15 -04:00
Domenic Denicola
ba1e7b956f More cleanup in conversion 2015-05-09 01:28:43 +02:00
Domenic Denicola
a26f622fbb Work on assembling the extras 2015-05-09 01:28:29 +02:00
Domenic Denicola
8bb41473d2 Throttle conversions; move to a separate file 2015-05-07 23:56:53 +02:00
Domenic Denicola
a4e5a12fe4 Use sortable filenames, and fix output filenames 2015-05-07 23:27:08 +02:00
Domenic Denicola
a820bc517b Don't fetch or execute JavaScript 2015-05-07 22:57:35 +02:00
Domenic Denicola
2236bb2a86 Try to parallelize the conversion more 
Stalls before it gets to any writing though, it seems.
 2015-05-06 23:08:59 +02:00
Domenic Denicola
127b6612c9 Re-architect 
Separate out download code from the rest. Re-do the download code to store URLs alongside HTML, and add support for resuming from the last-seen URL.
 2015-05-06 22:17:26 +02:00
Domenic Denicola
972fcaa294 Take care of another broken Next Chapter pathology 2015-05-06 01:30:55 +02:00
Domenic Denicola
1e346e7f9c Handle URLs that io.js can't handle 2015-05-06 01:22:04 +02:00
Domenic Denicola
45b5b9d6fd Restructure to be a bit more efficient 2015-05-06 01:16:27 +02:00
Domenic Denicola
925dcc6861 Detect "Next Chapter" links better 2015-05-06 00:58:52 +02:00
Domenic Denicola
ae1937df41 Lint better 2015-05-06 00:49:36 +02:00
Domenic Denicola
039c0fb2eb Got chapter downloading and conversion working 2015-05-06 00:42:55 +02:00
Domenic Denicola
3c2f67000e Initial commit 2015-05-05 22:47:45 +02:00