FYI
Originally shared by Sage LaTorra
Looks like the Dungeon World Codex is down. We got DoS-ed by what appears to be some kind of weird crawler. A Mozilla/Windows user agent hit every link it could find at roughly half-second intervals viewing every single monster page.
We don’t have crawling blocked and haven’t had a problem in the past, so I’m thinking this is just some rogue agent. It’s funny to imagine someone caring enough to DoS us on purpose, but I doubt that’s the case.
I’m going to keep it down for a bit until either our quota resets or I can figure out what’s up with this crawler.
When it gets back up I’ll be sure to post a new monster: “The Crawler” with the moves:
-Ruin everybody’s fun
-Hint towards malicious intent
– Do the bidding of some Jag-weed with no life.
Just to be clear here: the issue is not that the system went down, but that in order to serve out that many requests we’d have to pay money, which we don’t want to do.
How many entries are in the codex currently?
Sounds like someone scraping it?
Scraping sounds likely, yeah. Or indexing.
I can’t get an easy count right now since we’re over the data store read limit (which apparently applies to the admin console too), but it looks like well over 900. I’d guess 1,000+.
Hmm, Must have been the Sparkly Vampire I added Yesterday.
At one point I was thinking about writing a scraper, to compile all the monsters down into one big PDF file. I will cease thinking that now.
It looks like you are missing a robots.txt. That will be enough to wave off any well-intentioned crawler.
http://www.robotstxt.org/
Providing a compressed bulk dump would channel the bandwidth of those that want to scrape and reuse the content.
The well intentioned crawlers aren’t the problem. I checked the logs, the google and bing crawlers are no problem.
I knew it! No, I’ve been lost since the first post.