It also reads in the “domains” at start, so if some phish site gets temporary offline and gets removed from online-valid.xml, it will still be listed in the dansguardian file. (This to prevent a phish site from temporarly dropping all its connections just to get removed from phishtank, and then reopening the phish site)

With second level domains, I mean that a phising url like:
http://adsl-75-11-237-21.dsl.rcsntx.sbcglobal.net/.irs/stimulus.refund/0,,id=181665,00.html

is converted to:

sbcglobal.net

The reason of why im doing that, is that a phisher can set up like millions of sites aaaaaaa.host.com to zzzzzzz.host.com. He only needs to point *.host.com to his IP in his DNS, and then setting up a *.host.com virtualhost, and then sending a random one to each recipient. This makes the listing at phishtank ineffective, if I don’t block out the whole second level block. The only part the phisher dosen’t have control over, is the second level domain (that he has to purchase), and the TLD.

And here comes the script: http://pastebin.com/f2f4f0f27
You can then use wget to pull down online-valid.xml from phishtank, and then run the script that I have posted. Then you need to restart DG (/etc/rc.d/dansguardian restart) to reload the blacklist. Three lines of code (wget fetching, running perl script and then restarting DG) can be done from cron.hourly or cron.daily

I guess its a safeguard if Phishtank decides to change the format of the Phish detail url, but then it could be made that the current phish detail url format is specified in the metadata, only once per XML file, like:
and then the application developer only needs to replace %1 with the phish ID that the application developer wants to link/send the user to.

I think the “verified” and the “online” tag can be removed too, since they are constant.

There are some invalid characters in the URLs.
Have a look at phish IDs: 405396 and 360933

I have no safe way to know which entries were removed from the list. I toyed with the option of assuming that whatever is missing from the current file but was sent in a previous file is deleted, but since sometimes I may get a truncated or incomplete file (network issues, etc), I run the risk of accidentally deleting all the items that were sent previously.

Is there any way to make this more explicit – some kind of versioning mechanism like the Google Safe Browsing API uses?

Thanks.

Incidentally, it would help if the file was updated as an atomic operation. Occasionally, we see a partially written file, if we happen to read it while it’s being rewritten. We have to read the file twice at 30 second intervals and compare, rereading until we get the same contents twice in a row. It would be better to write a new file on each update, then move or link it to the name of the distributed file.

−
2007-10-12T04:30:01+00:00
0

That’s the entire file. Valid XML, no entries. Something is very broken.