Fileplanet closing down, archival in progress

Discuss anything not covered by any of the other categories.
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Fileplanet closing down, archival in progress

Post by qbism »

Spirit wrote:Any suggestions how to still make it public? Otherwise we could still save it but no-one could use it.
Expand contents to a web server. Use HTTracker or similar automatic clientside website backup tool to glean content that is exposed via links. Any files that are unlinked could be private. I have no idea if it would really work with this type of backup.
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

Nope, to do that we would need to crawl the whole internet to see what files are publically linked. And we would need a time machine to do that for all days from when they started hosting files. And also BBSs, mailing lists and IRC etc. It is a approach from the wrong side. :(

We have access to the raw file tree via FTP.

For Fileplanet it is different, we crawled that from the website end so we know what was public and what not.

Actually the FP staff used a metric like that to determine what files to keep (if I understood it correctly), files that did not get any hits in a timespan they checked will/are/might be deleted.

Well, archive first, think later. Maybe we can find a way to make it public in a good way.

I will personally grab a copy of the planetquake stuff and publish a file list. With a proper request and reasoning (proof) I could send out files from that then.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

This is taking so long. :)

We should have all files at the end of this month.

Re the private files. You could always download them when you knew the URL. How about we upload them at archive.org without them being searchable in the index? That would be the exact same way how they were hosted previously. Item names at archive.org will be created in a easy and readable way so that you can easily go from "original download location at fileplanet" to "item at archive.org".
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Fileplanet closing down, archival in progress

Post by Baker »

I just want to say that the archival efforts that you (and Chip and others) have done have been invaluable.

If all those FilePlanet downloads were permanently lost, that would be a rather large void considering how much was hosted there.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

I rented a bigger server (donate now!) so uploads are going live sooner now:

http://archive.org/search.php?query=sub ... eplanet%22

We got a "no" from the Internet Archive guys about uploading all the files seperately so you will get massive Tarballs instead. Copy the download link and append a / to browse them. I will provide a nice interface with direct links to single files some day.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

Getting really close to completion now for real. The 9.3 terabyte of public files (/ftp1/) are all uploaded to archive.org, they need some cleaning up though. The entry point will be https://archive.org/details/archiveteam-fileplanet (right now only some of the items are in there).

I do now have a copy of all the older files (/ftp2/) locally and they are also uploaded at archive.org (except a last one that is giving me trouble) but non-readable because of the afore-mentioned privacy issues.

I just verified the consistency of the ftp1 archives against a file list. Once I get the last ftp2 directory (stupid planetelderscrolls) fully downloaded, I will verify all of those too.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Fileplanet closing down, archival in progress

Post by Baker »

Thanks for all the work you have continually done (and LordHavoc too for QuakeDev and Chip's work too) to ensure that historical Quake mod source codes and other creations are not gone forever. There are tons of great and innovative things many different people did and ideas in those works. This game will likely be of interest even a 100 years from now just from historical reasons alone, but those aren't the reasons that interest me.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

We finished archiving a long time ago.

I am now finally working on a frontend. Very WIP but functional https://www.quaddicted.com/stuff/temp/f ... name=quake

Not fancy, but you can download fine.

If someone wants to collaborate, please shout. It's php & sqlite3. Great features to have would be a realtime search box and pagination. I have more metadata, screenshots etc, which I will join later. I will probably also properly restore the timestamps, noticed the "23:00:00" just now. I have them accurate to the minute.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
leileilol
Posts: 2783
Joined: Fri Oct 15, 2004 3:23 am

Re: Fileplanet closing down, archival in progress

Post by leileilol »

oh wow, you're an unsung hero on preserving video game patches as well.

and subscriber-exclusive betas! yay!
i should not be here
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

Thanks! :)

Yeah, there is nice stuff inside. I should add that the Direct2Disk (or whatitscalled) things in /dd/ are encrypted installers so don't waste your time downloading those (and don't speak about them).

I meant to work on it today, btu didnt. So, some more info:

This does not include the ftp2 stuff, that's the "hosted files" from the planets etc.

You can grab the db at https://www.quaddicted.com/stuff/temp/f ... .sqlite.7z

Please do not mass-download via this interface. If you are after big chunks, please consider downloading the big TARs completely instead. I am not sure how IA's tarview system would handle many hits. I still have to ask them that...

If I knew how to do so nicely, I would add a browsing interface.

Rather don't spread this yet, it's not meant to be public to "the world".
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
leileilol
Posts: 2783
Joined: Fri Oct 15, 2004 3:23 am

Re: Fileplanet closing down, archival in progress

Post by leileilol »

yeah I noticed that after it took some time to initiate a 100mb download of one file, I assume that's tar extraction on demand
i should not be here
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

I added listing all files from a directory (and its subdirectories): https://www.quaddicted.com/stuff/temp/f ... ion/quake/
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

10 years later I clean up this archive. Affordable huge hard drives for the win!

There are some files where there is a difference between the file grabbed when going by ID through the website and the file I got directly from the FTP. And for at least two files I could use some help. If someone is able to either check them for corruption in smarter ways than me or unpack/install them to see if they end up broken, that would be super helpful! The "byid-" and "fromftp-" prefixes are from me.

Project Reality installer (Battlefield 2 mod)

https://www.quaddicted.com/files/temp/f ... _setup.exe
https://www.quaddicted.com/files/temp/f ... _setup.exe

sha1sums:
af77021a2b78221465376687fb04d332ea97d74b byid-pr_0909_full_part2of2_setup.exe
3cc425ba77c8c5fe307845db36ecf2704354490d fromftp-pr_0909_full_part2of2_setup.exe

md5sums:
3c1af12bc746e77f38b9676adcfd53eb byid-pr_0909_full_part2of2_setup.exe
7833035590a24b92ab5a8168ff7d6ef9 fromftp-pr_0909_full_part2of2_setup.exe


Installer for Project Reality, some Battlefield 2 mod. The installer wants a Battlefield 2 installation, which I don't have.
This one has lots of differents bytes somewhere inside. Both launch as installers though. And both can be unpacked with 7z, resulting in 100% identical file trees from what I see (tested with diff -r -q).

The fromftp file matches https://www.ausgamers.com/files/downloa ... art-2-of-2 so that looks 99.9% like it is ok and usable.
Neither file matches http://files.realitymod.com/bt/pr_0909_ ... xe.torrent though.


Something for some Riddick game
https://www.quaddicted.com/files/temp/f ... _setup.exe
https://www.quaddicted.com/files/temp/f ... _Setup.exe

sha1sums:
1c55ba6e584d4c57448bd08897a0a613a5abd9da byid-riddick_setup.exe
ae722b6c862a8d432c0a15dfae0aa00be0a1127f fromftp-Riddick_Setup.exe

md5sums:
db5e304c7ac3c98d25091e2c062753f1 byid-riddick_setup.exe
ff871d91247a259defea2018f584f38c fromftp-Riddick_Setup.exe


This one seems to want an existing installation of Riddick, which I don't have.
The byid version has nothing but zero bytes from some offset onwards but I cannot be sure it is broken and the other intact without some proof.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

And two MacOS things with bad bytes. In both cases the FTP version has lots of null bytes in the places where the byid version has some other byte values.

It would be awesome if someone with MacOS could test them.

EU2UK_ddsetup.dmg
This one seems to have two bad bytes at different places.

dmg2dir says:

Code: Select all

ERROR: dmg image is corrupted
Command fails (error 10).
7z (version 21 too) prints a warning:

Code: Select all

Cannot open the file as [Dmg] archive
The file is open as [HFS] archive
So I have no idea how to successfully unpack them. In any case, I could not be sure they work on MacOS. Could someone test this?

https://www.quaddicted.com/files/temp/f ... dsetup.dmg
https://www.quaddicted.com/files/temp/f ... dsetup.dmg

md5sums:
af5c65e4a1d2850683d6bd9f58645670 byid-EU2UK_ddsetup.dmg
f5b461681c4770b3853f10034825fabf fromftp-EU2UK_ddsetup.dmg

sha1sums:
c9bd81b97ea59c3e13a2ff0356e73a3d1b88c23e byid-EU2UK_ddsetup.dmg
faca10ff4d84cf03073db601ad201fbe7f69baea fromftp-EU2UK_ddsetup.dmg


LOTRO-OSX-RoI.dmg DONE, the FTP version is fine
This one seems to have two bad bytes next to each other.

7z shows the following errors for both files but they are unrelated and seem like good errors ;)

Code: Select all

ERROR: Dangerous symbolic link path was ignored : Disk Image/Lord of the Rings Online.app/Contents/Resources/dosdevices/c_ : ../drive_c
ERROR: Dangerous link path was ignored : Disk Image/Lord of the Rings Online.app/Contents/Resources/dosdevices/d__ : /dev/rdisk1s2
ERROR: Dangerous link path was ignored : Disk Image/Lord of the Rings Online.app/Contents/Resources/dosdevices/e__ : /dev/rdisk2s1
ERROR: Dangerous link path was ignored : Disk Image/Lord of the Rings Online.app/Contents/Resources/dosdevices/z_ : /
ERROR: Dangerous symbolic link path was ignored : Disk Image/Lord of the Rings Online.app/Contents/Resources/English.lproj/MainMenu.nib : ../WineskinLauncher.nib
Comparing the extracted outputs there is zero difference. But I cannot be sure they both work when used on MacOS. Can someone test?

md5sums:
743cc61f6630d195fdce1502df957413 byid-LOTRO-OSX-RoI.dmg
1be3ac9819921cdcb178891d79476c88 fromftp-LOTRO-OSX-RoI.dmg

sha1sums:
1460723041f315138f030c53ad036c879956fce7 byid-LOTRO-OSX-RoI.dmg
192377c348b351f041ff46eb2c0a8072bc8e91d6 fromftp-LOTRO-OSX-RoI.dmg
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Fileplanet closing down, archival in progress

Post by Spirit »

https://twitter.com/ItEndsWithTens/stat ... 0630279171 suggests that byid-LOTRO-OSX-RoI.dmg has a broken checksum and fromftp-LOTRO-OSX-RoI.dmg is not. To me that is enough proof that the FTP version is the one to use.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Post Reply