I took a deeper look. It seems the process was reading and writing only two files for 99% of time - it was C:\Users\myuser\AppData\Roaming\Bitrot Detector\BitrotDetector.sqlite-wal which was 190 MB of size and C:\Users\myuser\AppData\Roaming\Bitrot Detector\BitrotDetector.sqlite which was 45 MB of size.
For a few seconds of monitoring, there were about 200 IO operations on real files with my data, and about 180000 of operations on these two files. It means, that the database operations are not very optimal... even sqlite supports transactions, and the program could cache some values before importing them to the database.

I am not impressed.

Rafal Stanilewicz, it has become painfully evident in recent days that Bitrot Detector has some major performance issues when scanning a large number of files. This happens whether those files are large or small. Part of the problem may stem from certain design choices that keep BRD's scan database size small (an important goal, given how quickly scan data could grow if this was not accounted for). Improving performance is the number one priority for Bitrot Detector's next release (which should happen within the next six weeks). The goal will be to keep BRD's database size small, but for it to scan with a quick, consistent speed regardless of the number of files covered by the scan.

All users of Version 2016 Release 1 will get a free, optional upgrade to the next release, including those who installed and registered today's giveaway. Bitrot Detector checks for updates once a week by default, and can also check for this update manually (using the "Check for updates" button in the About tab). Another option is to subscribe to Domador Software's mailing list (low volume) to be notified about this update.

Thanks for the detailed feedback.

therube, thank you for your feedback. Here are some responses to what your wrote.

Yes, there is currently no way to run a scan without a schedule. Most users would probably be interested in scheduled scans, given that at least two scans are needed for Bitrot Detector to be of any use in detecting file changes, and to have ongoing notifications of potentially-unwanted changes. I'll consider adding an option to turn off a scanning schedule, so that a scan profile's scans are on an exclusively manual basis (with the "Perform Scan Now" or "Perform another" button).

Displaying the full path on mouseover for the paths in the Results window is a great idea. Currently, you can click on one of the paths and the path will appear below, to the right of where it says "Selected item". The full path IS displayed on mouseover in that section, and you can also view it in Windows Explorer by clicking the Show button.

When you notice that a file has been changed or deleted, you can determine if you expected and/or wanted the change or not. If you didn't want that change, that'd be the point where you could look in your backup and restore that file. (Bitrot Detector isn't very useful if one doesn't already use a backup system of some kind.)

Bitrot Detector does not change any of the files it scans. The only files it changes are its own data files, primarily the database where it stores its file fingerprints and scan data, BitrotDetector.sqlite. That file is indeed an SQLite file, though an encrypted one (to help protect fingerprint data from potential tampering by malware).

Opinionated, Bitrot Detector indeed asks for the Windows password because it is needed (unfortunately) in order to create a scheduled task in Windows' Task Scheduler (in certain versions of Windows). This task opens Bitrot Detector at scheduled times, in case it isn't running already. The goal is to make sure BRD's scheduled scans aren't missed, but are performed at their scheduled times. Fortunately, newer versions of Windows don't require a password to create a Task Scheduler task. (You can try this out yourself by going to Start -> All Programs -> Accessories -> System Tools -> Task Scheduler. Try creating a new scheduled task. If it asks you for a Windows password at the end, then Bitrot Detector would also need to know the Windows password to create a Task Scheduler task.)

That being said, Bitrot Detector doesn't NEED to create a Task Scheduler task, or know your Windows password. If Bitrot Detector is already running (and your computer is also running), BRD will start a scheduled task at the scheduled time. But if it isn't running, it'll be up to you to open it up so it can perform your scans.

Seb, that's indeed how hashes work. A minor change to the input (a period in this case) will produce a large change to the hash. This might not seem so useful in the example you gave, but thanks to this, Bitrot Detector can compare ALL the versions of each file that it has ever scanned WITHOUT having to store a full copy of each and everyone of the old versions. For example, instead of storing a copy of a 650 megabyte file (such as .iso file for a full CD-ROM), Bitrot Detector stores 32 bytes (a 256-bit SHA-2 hash) which serve as a fingerprint for that file. Then, the next time it scans this file, Bitrot Detector will generate and store another 32-byte fingerprint (instead of 650 MB). Just by comparing those fingerprints, BRD can tell you whether the file has changed, regardless of whether the change is a single bit, several bytes, or many megabytes. (It doesn't tell you *which* part of the file changed, or how much it changed, but for that you can compare the current copy of the file with the one in your last backup.)

Different kinds of programs are needed to see the actual differences between two files of the same kind. A text editor is needed to compare text files, a photo editor to compare photos, an audio editor for sound/music files. However, Bitrot Detector can detect any changes to any of these different kinds of files, and one can review those changes (and see if they're undesirable changes) with the corresponding program.

Andres Cabezas,
It needs a huge amount of complex development work to make it a really useful program, probably AI..
In fact the way Windows works makes it pretty well impossible to be useful without a huge amount of input by the user.
I agree with your description of how it functions but do not see it as a practical and useful program.
My granddaughters a Cabezas. :-)

Does anyone know of a WIndows Explorer addon (Comes up when you right-click a file in windows explorer) to generate a file containing SHA-2 Checksums... All the selected files will have a checksum generated into a txt file. The one I use can only do MD5, SHA1 or SFV.

Mememe, Bitrot Detector is aimed at a different kind of use than you have in mind. It looks like you'd like to generate, view, and store SHA-2 checksums for individual files. Though Bitrot Detector generates and stores them, it doesn't currently display them, though that may change in the future.

I looked at the tools in the link you provided. They seem very useful. The main difference between them and Bitrot Detector is that they're all real-time monitoring tools. They'd need to be running in order to detect that a file or folder has changed. Bitrot Detector is more of a "snapshots in time" tool. It can detect that changes took place, regardless of when such changes happened, and even if Bitrot Detector wasn't running at the time the changes took place.

Realtime monitoring of watched folders is a potential, future feature, though there are other development priorities for now (such as improving Bitrot Detector's scanning speed).

TK, right now that won't be possible since the SQLite database is encrypted. The ability to export hashes will likely be added to an upcoming Bitrot Detector upgrade.

Rafal Stanilewicz, thanks for your feedback!

There is unfortunately no way to determine whether a file on a computer was changed interactively by a user or changed by a program operating on its own. A program on a computer can't tell whether it is being used by a human or used by another program to edit data.

Due to its underlying design, Bitrot Detector can't currently determine how much data was changed in a file, just that the file's contents changed in some way. Bitrot Detector stores a 32-byte fingerprint of a file's contents on each scan, and compares those fingerprints to determine if changes have occured. In order to determine how much a file has changed, BRD would have to store a copy of the original file, and a copy of the changes detected on each scan. This could use up a lot of space on the hard drive, whereas by only storing fingerprints, BRD uses little space. However, this feature is worth considering, and maybe users could have the option of determining which files BRD should store a copy of in order to perform detailed comparisons (such as the %age that has changed.)

BRD already detects changes to modification dates and file sizes. (That data is also collected, along with the 32-byte fingerprints, on each scan.)

I'd like for Bitrot Detector to be as clear and easy to use as possible, so I'll definitely look into ways to improve the scan status/statistics window. I'll also look into the tutorial as well.

Thanks again!

nomadss1, I'm not sure what you're referring to, as far as having to type a lot. Maybe you're referring to the "2: File types" tab in the "Create a scan profile" window, where one can select specific extensions to include, or type in the extensions manually. However, in the first tab ("1: Folder"), you can click on the "Add a folder" button to select a folder to include in the scan (by clicking around). You can include additional folders by clicking on the "Add a folder" button again. Also, if you don't want to select a folder individually, you can check the "(My) Documents" line which is already displayed in the first tab, and scans will cover anything in the My Documents folder (and any subfolder).

nomadss1,
Equally unsure as to where the typing is necessary, it allows used to select folders when you're setting up a profile using a folder browser.
Not so much "sloppy programming" as sloppy reviewing!

Lenny, thanks for trying out Bitrot Detector, and for taking the time to provide feedback.

Yes, Bitrot Detector is a bit bland right now, and bare bones as far as help files and tutorials are concerned. With every bit of feedback, I'm getting a better idea of what kind of supplementary information may be needed to help everyone make the most out of Bitrot Detector.

The results you describe are indeed disappointing. Just to confirm: did you select the WHOLE hard drive for scanning (for example, C:) as opposed to just My Documents or your Windows user folder? Assuming you did, Bitrot Detector should have scanned the Program Files folder along with the rest of the har drive, and detected the little used program the first time around, then detected its absence the second time around. BRD did not consider the new programs you installed to be "significant changes", as you're adding new data to the drive, as opposed to deleting or modifying existing data. (It's easier to recover from "extra" data than from missing data.) It's worth considering adding the option for BRD to explicitly notify users of additions to certain key folders (such as Program Files) and to consider these additions significant. However, the new files installed by the two new program should have been detected and listed in the "All items: New" category in the scan results window.

If BRD didn't detect the little-used program you later uninstalled, there may be some permission issues to investigate, to see if Windows blocks Bitrot Detector's access to certain files by default.

Brian Clark, This is said with tongue in cheek since I've never used "Bitrot Detector", however, by reading the description, both here and elsewhere, if you do your due diligence with "Bitrot Detector" or any other program of it's type it could be one of the more valuable programs on your computer.

I concur; many programs, on any computer, are modified not only on a daily basis, but some on a regular basis, minute by minute, hour by hour, however, if one does their due diligence with this program it could be quite valuable. Is there another program, of it's type, that does it better? I don't know; never used a program like this before.

You could for example eliminate all .txt files from the scan after your first scan because no malicious program I've ever seen modifies .txt files, although some do add .txt files and it would be nice to know if a .txt file has been added in addition to another program either being added or an existing one being modified.

What I get out of your comment is that it's too much bother to use this program. If that's the case then don't use it, however if one does their due diligence with a program like this it could prove to be quite valuable.

Glen C, No similarity between "Bitrot Detector" and "SpinRite"; SpinRite checks and repairs the physical aspects of your Hard Drive while Bitrot Detector checks files on your Hard Drive for modification. No similarity between the two what so ever.

"SpinRite", in my opinion, is the top program of it's type. I haven't used "Bitrot Detector" yet and so can give no opinion.

JonE, Thanks. I snapped after I woke up.

Glen C, The solution you're describing (CrystalDiskInfo + SpinRite) detects low-level hard disk damage. Bitrot Detector is aimed more at the file level, to detect (potentially unwanted) changes to individual files. On rare occasions, these changes will be the result of a damaged hard drive, but usually they'd be caused by accidentally deleting, moving, or editing files. (Such changes could also be the result of viruses and malware.)

Seb,
I think the use is this. If you usually back up say your photos to another disk by mirroring all the files. Then at some point unknown to you some of your original photos are damaged (this has happened to me). When you do another mirror both your orginal and your backup are the damaged version - thus photo is lost. This program may inform you that damage has occured either to the original or the backup so you can restore the damaged files before both your copies have the same damage.

David Murphy, this is precisely one of the common situations where Bitrot Detector can prevent data loss. With mirrored file backups, all the files that are present *right now* will be backed up. However, any files that were deleted *since* the last backup would be lost when a new backup is performed. In the time between each mirrored backup, Bitrot Detector would tell you which files have been deleted or changed, giving a user the opportunity to go and copy the file from their last backup. This way, it will be included in the next backup. This is one of the kinds of subtle data loss that Bitrot Detector seeks to prevent.

TK,
Totally agree, my present OS is five years since installed and I'm starting to see file corruption.

Unfortunately, this program can only report on file corruption it cannot do a recovery, very little use in reality as far as I can see.

Feel that if it was a really useful program it would have a very good explanation and help file, it has nothing suggesting the programmers are not really sure of its usefulness or function.

Seb, use par2 or multipar to create recovery archives for your files that you know you only want to store (eg. pictures and such). They can be used to recover files when data corruption happens.

Just taking snapshots of a file checksums and telling the user that there has been corruption or change to the file is utterly pointless if there is no option to recover the original file.

David Murphy,

That reminds me of a program called crc32. Which creates crc numbers for each file so the user can check to fibd out if tge file has been changed.

TK, detection of the deleted photo by Bitrot Detector would take place the next time that it scans the folder that contained that photo. Bitrot Detector doesn't currently detect changes in realtime. It might do so, eventually, to help keep Bitrot Detector usage from becoming tedious due to false positives.

I really appreciate your comment about false positives. That could definitely be a drawback. One additional countermeasure, at least in the case of photos, would be to store thumbnails, low-resolution versions of the photos which don't take up much space, but which could help a user quickly determine whether to be concerned once BRD tells them a photo was deleted. Creating such thumbnails could be processor-intensive, so such a feature would need to well thought out. Among other things, it'd need to be an optional feature. If it eventually makes it into Bitrot Detector, it probably won't happen for several releases.

Regarding bitrot, yes, literal, actual bitrot is rarer that other causes of data loss, but I still wanted to use that term as part of the program's name. Like you, I also wish Windows used and natively supported file systems that have a greater focus on data integrity, something like ZFS. Maybe someday...