Bug:137

Q&A

 * What is the approximate frequency of the bug reports now?
 * April 2014 (2.0.5 current): Reports that look unequivocally like Bug 137 (data files moved or missing) are now fairly rare, probably about six or seven reports a year.
 * There have been three recent reports against 2.0.2, 2.0.3 and 2.0.5 on Mac OS X that show projects reopening with missing block files.
 * Two of these cases occurred where multiple projects were open simultaneously when saving the project. The 2.0.5 user encountered missing block files "frequently" when reopening projects after pasting audio between open projects. Unknown if cut or copy was used.
 * In addition to missing block files, the project created in 2.0.3 displayed the "Problems reading sequence tags" message when reopened and the log showed "Start for block file is more than one sample past end of previous block" errors.
 * There are still very occasional reports of AUD*.temp files created in the user's folder for audacity-data (these are distinct from the .tmp files in the "AutoSave" folder which can be left behind if Audacity cannot write the autosave file). I have seen these AUD*.temp files appear in the audacity-data folder myself on Windows 7 in 2.0.3 after running the shipped "MP3 Conversion" Chain on files, but this did not destabilise Audacity. I can't reproduce the problem and have not seen it again.
 * Loss of the AUP file, leaving only the _data folder, is reported at least once or twice a week on the Forum (usually on Windows). Many of these are believed to be user error, but if there is no check that the AUP file is written and is legitimate XML, it is unsafe to assume these are all user error. This comment is an aside for information, as this problem is best dealt with separately.
 * The AUP file apparently being saved at correct size but just containing NULL's is reported once every two or three months.
 * Was 10-12 per month (reported) with about half losing data. Since 1.3.13 about 8 - 10 per month (legitimate reports) with about one-third losing data, but with an apparent slow decline in frequency.
 * James The (previous level) sounds too high a level to call it 'comet-returns'.
 * What is the OS?
 * Something like 50% Win, 40% Mac, 10% Linux, with Mac % having gradually gone down over time from being on a par with Windows.
 * Is this associated with closing Audacity whilst Audacity is doing something - such as updating auto-save? (wild speculation from James)
 * No evidence of that. How long would autosave take on a 3 hour project? Doesn't the quit wait for autosave to finish?
 * Presumably detecting orphans does not lead to silencing, but detecting missing block files does?
 * Where Audacity has moved .au files between simultaneously open projects then orphans in one re-opened project will be missing audio in the other project. So deleting the orphans will permanently remove the ability to restore silenced audio in the other project.
 * Which fixes appear to have helped the problem?
 * The UTF-8 ones??
 * Gale: Passing .aup files between platforms is rarely mentioned in the bug reports. I have a hunch (unfounded, but it ties up chronologically) that the fixes to prevent unwanted zero length clips appearing may have helped.
 * Should orphan checking be at the subfolder level to catch more problems, rather than assuming as long there are no .au files not referenced in the .aup that the entire structure is correct? Missing file checking seems to be at the subfolder level, so theoretically Audacity should be able to check all folders in a project for a missing file and if found, put it in the correct subfolder indicated in the .aup.
 * Won't help files that are in the wrong project. Do we know that files are being moved into wrong sub folders?
 * Gale: Yes, several confirmed reports that happens, and I saw it happen myself as per the Summary.
 * James: That suggests the developers have made a nooby error, relying on current path when opening files rather than giving full path. If so, that would be very embarrassing, but easy to fix.  It could also be a more subtle error with passing file handles between threads, which is a no-no because wxWidgets objects are typically not threadsafe.
 * What are the circumstances under which multiple "d*" folders are allocated inside the "e00" folder? I added to the release note that "having a long project with many tracks could cause the issue to occur" because I think this makes it more likely there will be multiple d* folders and this raises the risk of Audacity misplacing files. I've reviewed cases where people report cutting at the end of tracks as a problem but many are not quitting and restarting so I suspect they are probably seeing comment 23.
 * Projects with large numbers of blockfiles will have large numbers of d** folders under the e00 folder. Generally large numbers of blockfiles means large projects (high sampling rate, long duration, many tracks).  It's also possible to temporarily have a large number of blockfiles by doing a lot of edits, since the history keeps them until you close the file.

Oct 2011 interesting Forum reports
The following are very recent reports from early October 2011 where there seems to be data loss and grounds to suspect a bug (as opposed to reports of orphans and data loss where there are possible grounds to suspect user error):
 * http://forum.audacityteam.org/viewtopic.php?f=16&t=61262
 * http://forum.audacityteam.org/viewtopic.php?f=16&t=61258
 * http://forum.audacityteam.org/viewtopic.php?f=16&t=61223

Plus the following report showed thousands of unexplained AUD*.temp files created in the user's folder for audacity-data leading to crashes changing Tools:
 * http://forum.audacityteam.org/viewtopic.php?f=16&t=60999

User denies that Audition or some other program created them.

Orphans on Sleep/Hibernate

 * (user to feedback@audacityteam.org 2011-April-7) "I'm using Audacity on a notebook with Win XP SSP 3. I normally do not shut down the notebook, but use the sleep function. After two or three times sleeping (while Audacity is opened) i can store Audacity .aup projects correctly, but if I want to open a project, Audacity finds a lot of "needless" files and proposes to delete them."
 * James Comments: This suggests a repeatable way to get Audacity to create orphans. Timer related?
 * Hmm, does he save the project before sleeping, or rely on auto-save?
 * Gale: I'd love it if sleep or hibernate led us somewhere, though as I told him I have never found a problem so far with either sleep or hibernate. I am not following about "timers". I thought we had ripped that out. AFAIK every change to project state is saved in the autosave file when it happens (i.e. when the files in the _data folder change).
 * James: I think you're right. Timers as a theory won't fly.
 * Bibek: I can confirm that I treat my laptop the same way (Vista SP2, 32-bit, 3G RAM), sleeping rather than shutting down, running Audacity 1.3.13 beta, and have seen the same problem. I haven't tracked it in as detailed a manner as the other user, but his description seems to fit my memory.  I thought I had a set of files here somewhere that were botched up like this, but I can't find 'em now.  If I at some point stumble across this again, I'll see if I can create a directly-reproducible test-case.
 * Gale: Does the computer sleep when the project is unsaved (File > Save Project is active)? Or is the project open but saved, or is the project closed but Audacity running? In 1.3.13 Beta you will get orphans when you cut to the Audacity clipboard, close the project without exiting Audacity then reopen the project. The orphans disappear when you exit Audacity then reopen the project. That is fixed in 1.3.14 alpha (the current "Nightly Builds").
 * Bibek: Also, I believe Windows supports some hooks related to sleeping / waking, so is it possible to add a sleep / wake hook in Audacity that just logs information about the state of the program to a file, in order to help debug the issue? The log may show an oddity even if the program appears to be running normally for the developers.

File read/write errors

 * Ed June 2011: I would be willing to bet at least a nickel that the problem is involved with not (properly?) testing the return value of a disk write procedure.
 * Gale: I think write errors are a credible hypothesis. There is limited evidence that this problem tends to occur on less well resourced machines, even a couple of suggestions that installing new RAM or hard drives coincided with the problem not happening any more. If an .au file can't get written for any reason, will it still be referenced in the .aup file, so creating a missing file? I understood (maybe wrongly) that the .aup file gets written after updating the .au files. But if the .au file gets written but the .aup file does not, then you have an orphan that the project actually depends on.
 * Ed July 2011: The .au files are all written to disk (ignoring any errors) then the .aup file is written to disk using the assumption that all the .au writes succeeded. An .aup would revert if write failed but with no warning! It seems _data changes are not reverted so this could easily lead to an AUP being out of step with its _data folder. If user was copying from one Project to another, it might even exhibit as AUs in the wrong Project. But implementing and testing these changes will require months of field testing even if they were the root cause of Bug 137.
 * Gale July 2011: I had a small project "E:\16.aup" that was just a number of tracks with short recordings, undone back to the first couple of recordings. I could edit and save the project at will but "Save As" failed to any arbitrary directory with an "unable to write - perhaps full" message. This message on "Save As" has been reported a few times against 1.3.13 with no apparently legitimate reason for it. Copying the tracks to a new project and "Save As" in that project saves correctly. However at least in my case, the original project I was trying to "Save As" was corrupted in the process with many orphans and missing files.
 * Gale Sep 2011: This scenario has not been reported for a couple of months.
 * Gale Sep 2011: The Windows Unicode Nightly Build has had Ed's WriteReturns3 patch for a few months which automatically puts Write (& probably Read) errors in the log. There are also message boxes when there is an error (whether there is a saved project or not). To date this has not produced any useful feedback. I have not seen any log messages or message boxes to date although I have had project inconsistencies apparently due to moving/deleting .au files when re-opening.

Stress Testing

 * Michael tried to script this in April 2011 and got as far as getting a project to open from script by adding new command, but there were too many remaining issues for him to script. He did many edits with multiple projects and couldn't replicate anything.
 * It is possible to run Audacity with small blockfiles by launching it with the -blocksize option to try and provoke the bug. Michael did not test with small blocksizes. Gale has done so but it has not appeared to provoke the bug,
 * Ed has scripted writing to many projects when processing a large batch of files using a slightly modified "unattended CLI" Unicode Release. He has encountered about 5% errors in projects even with a very fast and powerful machine. He has found that when problems occurred, the occurrences were clustered. He found that starting a Norton full scan while the script was running always caused missing blockfiles on one machine. He also found occasional errors if Explorer windows were watching the project or Audacity temporary folder.
 * Gale: I always watch Audacity projects in xplorer2 windows.
 * Gale: All my failures are in Unicode Release. I do not regularly use Unicode Debug but have never had a Bug 137 instance in it. I would have expected at least a few errors based on frequency of use.

Past Investigations/Actions

 * James pointed out in Code Review Triage that there is a FIX-ME in DirManager.cpp(843): "Might we get here without midkey having been set?" This function rebalances directory trees and needs very close scrutiny.
 * He's now had a look at it, and midkey always will be set. The rebalancing function still needs closer scrutiny, as when it 'fails' it fails silently, rather than reporting an error.
 * Gale 01Jul11: There was a fixed bug 409 where the Audacity _data folder was being deleted when empty. It was agreed that deleting  empty subfolders was legitimate and being handled correctly, so not relevant to this bug.
 * There were some reproducible problems caused by UTF-8 characters and compression leading to missing blockfiles. These appear to be a (a) a separate issue, and (b) sorted now (by Michael).
 * Cutting audio places audio on the clipboard and these files were being wrongly detected as orphans when reopening a project in the same Audacity session. This is now fixed in r11118 with a follow-up fix in r11171.
 * "They should be deleted to avoid disk contention" wording in the orphan dialogue removed at least until the bug is closed (if reinstated afterwards, it should not mention "contention" which is about multiple process access). "Delete files immediately" changed to "delete files permanently".

Bugzilla Extracts
We won't always paste extracts, due to redundant work involved.