The Cat or Kitten
Evaluation / development version. Requires Net 2.0, see Sissle below.
This only requires unzipping the zip file to a directory and registering the dll by
executing the following in the directory containing the program and specifically the S3Extension.dll.
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\regasm S3Extension.dll
possibly the v2.0.50727 could be different
Note the reverse is
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\regasm /u S3Extension.dll
before you delete everything.
Options are under Help menu for entering Amazon S3 codes.
Download The Cat v 3.2(0.5MB)------ Mar 28 2007 13:00 CET
Sissle
Sissle requires no special installation, but does require Net 2.0 (go to bottom of page for download links). Unzip the Sissle.zip to any directory c:\program files\sissle for instance, right click on sissle.exe and produce a link. Copy the link to your desk top.
Download Sissle v 0.992(0.5MB)------ Apr 23 2007 16:30 CET Latest
Old encryption using RC4 is unaffected.
Fault in new Encryption.
There was a fault if for different files you specified different types of encryption.
If you have specified a particular encryption type and stuck to it then all will work, but it is necessary either
Before you do anything, note all security details.
1. Recommended. Download and upload with the latest version.
2. Poor man's solution. Note which files this applies to along with security details password, initial vector and encryption type and when dealing with these files ensure these are the values selected in the security dialogue. The date shown against each file shows the date uploaded so note when you update to the new version.
I followed what I had done for RC4 which was encrypted or not but of course with a choice of 4 new types of encryption I must store at least the type of encryption with the file.
This is now fixed. There is now metadata specifying which encryption is used.
Having made this error I analysed a bit more thoroughly the whole process and came up with the following:
- For Each encryption type you can enter a different set of security details. Before I only supported one password. Sorry more details to record.
- There is an option where the password and initial vector can be stored with the file. Obviously encoded. This is for people like myself who have a fear of loosing passwords.
This means that the encryption is now controlled solely by amazon security codes. A file will be encrypted on S3, so it can not be read by Amazon or the public if they have access to it. BUT if someone has your Amazon S3 access codes they can read it. It is not a full solution but it suits me.
- The final option is that no security details other than the encryption type is stored along with the file. Now someone must have your Amazon Access codes (unless it is public) and file encryption security details.
In this case the password and initial vector are not stored. So it is as I have said before important to know the values of the initial vector and passwords. If at some stage you change your password you have the added complication of needing to know which password applies to which files.
Encryption is good but there are dangers.
Download Sissle v 0.991(0.5MB)------ Apr 18 2007 16:00 CET
Fault fixes.
Corruption of Upload ReadWriteTimeout for files greater than 10 MB.
Improvement in abort of upload transfer.
Download Sissle v 0.99(0.5MB)------ Apr 16 2007 14:30 CET
This version extends the types of encryption that are possible. See security tab in options for possibilities. Please ! remember all passwords etc that you use, because if they are lost so are your files.
In earlier versions only small files (an option parameter) were compressed in memory. This is now changed to only small files (an option parameter) are compressed in memory, but larger files are now compressed to a temporary file.
I performed some tests wth other S3 products and wished I had done this early. To my embarressment the failures were not with incompatibilities on the S3 side but with the simple fact that several products mapped S3 to local devices with " in their names which in Sissle threw an unhandled exception and died. The products I tested were S3Drive and Jungle Disk and before there is any misunderstanding these are very good products and any fault was on my side. A few minor alterations were necessary to access some of the other good products that I tried, Firefox s3 and NS3.
I feel embarressed to say that if on the local side you select a mapped drive such as S3Drive or Jungle Disk and on the other side which is in S3 select another bucket and perform a transfer you are not cheating the system, the file is downloaded and then uploaded.
A context menu, right click and select from menu, has been added performing the most important functions. In particular creating bucket and making a directory allows larger text boxes for ease of entry.
Some work on the folder tree structure dialog to make it more usefull.
Download Sissle v 0.98(0.5MB)------ Mar 12 2007 17:30 CET
Sissle 1. evaluates all the steps in an operation and then 2. actions them.
Versions 0.96 and 0.97 were migrations to multi threading to improve step 2 above.
This version has made improvements in the first step and also cleared a lot of small faults in the program.
Download Sissle v 0.97(0.5MB)------ Feb 21 2007 15:30 CET
Version v 0.96 introduced threadpool action to uploads and downloads only. This has now been extended to
Deletions on S3 side of files and folders.
Setting ACL's of files and folders.
Synchronisation operations.
List Bucket and list remote folders
The issue on aborting with more than 10 threads has disappeared, I'm pleased, but knowing what the problem was would have been more satisfying.
So under Options you can now set the number of threads that sissle operates with, after setting the value you must restart the program The value is restricted to 1000, this is not so much a problem with the threads but each must maintain its own connection, so it depends on your machine etc. 10 is the default, 100 is slightly faster and it is difficult to tell if 500 offers any improvement. Maybe what follows overrides everything anyway.
I also have the following patch to Windows XP SP2, I give the details, whether you apply the patch is your decision.
SP2 introduces restriction see the following note (look under tcp/ip)
http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/sp2netwk.mspx#E5AAE
XP SP2 limits your number of half open connections connections to 10 (I have patched mine to 50)
I got the patch from
http://www.lvllord.de/?lang=en&url=downloads
or search google for EvID4226Patch223d-en.zip
There was a fault when aborting the operation of transferring files from S3 to PC in that the abort process could / did leave truncated versons of the files on the pc, this has been cleaned up by deleting files which have not completed the full process (download of full file followed by MD5Hash check).
Maybe a design fault, if you set a filter *.cs it remains set on subsequent operations until cleared. I have been caught wondering why no files have been found for operations until I have cleared the filter.
Download Sissle v 0.96(0.5MB)
Significant update.
1. Sissle multi threading issues.
2. More timeouts.
3. Include / Exclude filter eg *.cs; *.ico; *.rc; (; for multiple filters) on selecting files
4. Logging operations to disk. Errors to error.log and all operations to operations.log.
Multi Threading Issues
Initially Sissle created a single thread for S3 Operations. When transferring a lot of small files the performance was poor. The following is the historical migration to multi threading working, as is normal the most significant step is the last one.
Threadpool
Sissle transfers folders and their contents. This lends the code to be recursive in nature. This makes it difficult (not impossible) to utilise multiple threads. Most of the code has been rewritten for Upload and Downloads (only in this version) so as to be able to utilise multiple threads better. S3 operations are now run in multiple Threadpool threads. The difference was marginally better.
Multi Threaded apartment
Previous versions of Sissle were multi threaded in a Single Threaded Apartment (the default). This meant that though multiple threads existed they were executed in sequence rather than in parrallel. This was a bit embarrassing to realise that even though I could create 1000 threads from the ThreadPool they were processed one at a time to completion. Sissle requires the GUI to be in STA as it uses Interop services to access shell32.dll but now spawns the threads in MTA mode. Slight improvement. My knowledge in this area is not great and maybe some information here is incorrect, so back to safer details!
HttpWebRequest limitations
HttpWebRequest has a default of 2 active open connections, so even if I created 20 threads only 2 would be active and the rest would be queued inactive. I increased the default from 2 to 10. The performance was amazing, on a single thread transferring 200 small files took 135 seconds and now with a threadpool of size 10 it took 20 seconds.
A current thread " Download rate in Europe... " specifies that here (Germany) has a download speed of 410 KB / sec per connection. I can now fully use my home bandwith 6MB if I select a number of similar sized files to download.
Abort
Sissle is a GUI program and supports aborting the current set of active operations.
Previous versions actual did an Abort on the thread, actually I had no problems with this, but maybe I was lucky. Now events are set, detected, answer event set, etc, the outcome of this to the user is the the abort process takes more time, typically a few seconds. Unfortunately I have an issue that if the number of used threads in the threadpool is greater than 10 the Abort process leaves some of the threads hung and the only recovery is to restart the program. The problem is only with the abort process and the transfers work fine. This version therefore only spawns 10 threads into the threadpool.
Timeouts
For C# the HttpWebRequest implements 2 timeouts, Timeout and ReadWriteTimeout.
ReadWriteTimeout is the newest and easiest to understand.
The ReadWriteTimeout property is used when writing to the stream returned by the GetRequestStream method or reading from the stream returned by the GetResponseStream method. This timeout ensures for a large file that data is streaming in or out in a healthy manner.
Timeout is more difficult. From Microsoft
"
Note The HTTP request is made up of the following parts:
1. Sending the request is covered by using the HttpWebRequest.Timeout method.
2. Getting the response header is covered by using the HttpWebRequest.Timeout method.
3. Reading the body of the response is not covered by using the HttpWebResponse.Timeout method. In ASP.NET 1.1 and in later versions, reading the body of the response is covered by using the HttpWebRequest.ReadWriteTimeout method. The HttpWebRequest.ReadWriteTimeout method is used to handle cases where the response headers are retrieved in a timely manner but where the reading of the response body times out.
"
If you don't understand what this means, as I didn't, then select a medium size file and conduct experiments and you will arrive at the following.
This means that Timeout requires different values for uploading and downloading the same large file (ignoring the fact that most people have different up/downloading speeds). The following table simplifies the differences.
Transaction = header(Get + ...) + body (a file)
| HttpWebRequest |
Upload |
Download |
| Timeout |
transaction |
header |
| ReadWriteTimeout |
body |
body |
Sissle has 4 specific uses of S3
1. Upload files
2. Download files
3. Download in split mode (newest recovery from point of failure, instead of repeating download)
4. GUI which is predominantly List Buckets and List Bucket Contents. But also includes some odd/sods such as reading and setting ACL's and reading metadata. ie Shorter operations !
Sissle uses the following different timeouts (Options / Timeout tab) all in millisecs
| HttpWebRequest |
Upload |
Download |
Download SplitMode |
GUI |
| Timeout |
infinite(= -1) |
25000 |
25000 |
15000 |
| ReadWriteTimeout |
10000* |
60000 |
10000 |
10000 |
* Upload ReadWriteTimeout = file size > 10 MB ? value * file size in MB : value;
* initial algorithm, very likely to be developed
On top of the above is an exponential backoff as the number of consecutive operations fail.
Sissle now uses threads from the threadpool and as such had to be more robust than previously and I had infinite values for the download timeouts which could / did block the threads under certain conditions.
Transfer Window
The transfer window still displays which file is being processed and its progress, THESE VALUES ARE OF COURSE RUBBISH in a multi threaded enviroment. Next version should ......
Download Sissle v 0.95(0.5MB)
Repeat / Recovery For Large Files
Note. The HttpWebRequest AddRange(int start, int end) as you can see uses int parameters so restricts the values from negative 2,147,483,648 through positive 2,147,483,647. This restricts the size of files which can use this facility to 2 GB, my thanks to microsoft.
1. All testing had been for failures during transfer. A fault in the mechanism has been fixed for a failure right at the start of the whole process, that is no data had been received.
Testing of failure recovery in a reliable system is not easy, and has been the subject of at least one thread in the S3 forum.
The transfer status window
Did anyone find the new small button with a T on it. I used this to toggle failures within the program, for testing, and obviously should have been removed / disabled before issuing the program.
The listview columns are now sortable, size is a particularly useful one, when selecting / deslecting what to transfer.
Listview / Treeview checkboxes
I have made more tweaks to the listview / treeview checkbox interactions but it is clear that even more will be required. I am spending far too much time in this area and I must give serious consideration of redoing this section. In fact this must now be my top priority to sort out.
Download Sissle v 0.94(0.5MB)
A significant release. Repeat / Recovery For Large Files
1. The normal failure mechanism involves a certain number of repeats. For downloads the repeat recovery takes place from the point of failure, the download is not repeated from the beginning.
2. For totally failed downloads, there is now an error recovery mechanism which can be applied at a later time. That is if a download fails on 50% then it can be completed the next day (even after a reboot) starting from 50% (not the beginning). Use Options / Retry Download, this will present a list of failed downloads and their status, select a file to recover, click ok, a more detailed box will show more information, typically how much has been received and the files total size. If you are happy with this information, click Recover and the file will be downloaded from the point of failure.
The splitsize is the amount requested and downloaded in a single transaction with S3, if there is any failure then the download is restarted by requesting the failed section. There is a judgement on the value assigned to splitsize, for instance if your files are typically 1.0 GB then a splitsize of 20-100MB can be justified. There is a time cost (and a small bandwidth cost) in downloading a file in say 1000 sections as against in 1 lump. If the splitsize is greater than what you download in about 1 minute then the extra time is hardly noticed. Each user should conduct his own experiments here to find a value or set of values that apply to his needs.
This mode, like the original mode, now also supports MD5 checking, encryption and zipping, under the same conditions. Progress bar problems also fixed in this mode. Advice
This is only for large files, also a small split size of say 1KB which I used for testing has a vast impact on download speed.
This mode is not default, so in Options / Transfer Options set "Split Downloads" and adjust "Split Size"
For "Split Size" select a size you typically download in 1-5 minutes.
Amazon S3 unfortunately only supports this on download and there is no upload version.
If you wish to test this set splitmode and splitsize to 1024 (silly value but good to test with) and then download a file of say 100KB.
Part way thru hit Abort.
Let the program fail gracefully
Then use the Recovery procedure described above.
You can fail the same download several times, but beware that the failures are artificially created at your end which will cause Amazon S3 ( I believe ) to apply an exponential timeout, my experience is that after about 3 quick failures 1. make 2 cups of tea or 2. stop and immediately start the program, a new connection does not have these problems
Download Sissle v 0.93(0.5MB)
Fixed a fault in on the fly unzipping, a small section was added to the end of the file.
It also adds the following
Download Sissle v 0.92(0.5MB)
As v 0.91 was an emergency release, this is a follow up, more thoroughly tested version.
It also adds the following
Download Sissle v 0.91(0.5MB) I had been wondering why I had zero feedback and I think I now know why. I had introduced an horrendous fault dealing with multiple selections while transferring. This is an emergency fix as the first version is nearly unuseable. Ideally I would like more testing time, but even that does not ring true considering what I did with v 0.90.
This is a problem with the display of operations and can be bypassed by
Options / Transfer Options and check "No transfer Window - Just Do it"
if any further problems.
Other changes
- Only numbers allowed in numeric boxes
- Option of setting the upload and download buffer sizes
Download Sissle v 0.90 (0.5MB)
Net 2.0 (23 MB)
Download Net 2.0
or
if you prefer the official version or require another language other than English use-
Microsoft Site For Net 2.0
|
