Editing audacity.sourceforge.net

How the site works
In order to make the site available in many languages, we use PHP and the GNU gettext tools to deliver one set of pages translated into many languages. This makes it possible to update all language versions of the web site with minimal effort. Text that has not yet been translated into a given language will appear in English until the translation is updated.

The site has a single set of PHP pages which contain the English version of the pages. All text that appears on the screen is stored as PHP strings (rather than using included HTML) and output using =, echo and similar. These strings are marked for translation by enclosing them in brackets preceded by an underscore. _('some text');

When the pages are altered, xgettext is used to go through all the .php pages on the site and construct a message catalogue of all the strings marked as needing translation. Duplicates and known standard strings (dates and so on) are removed at this point to create audacity_website.pot, which is the translation template. A copy of this template exists for every language the web site is translated into, named after the ISO language code with a .po extension (so fr.po, de.po).... These contain all the strings marked for translation and the translation into that language of the string. These files are maintained and updated by the team of Audacity translators for each language.

Once translations have been obtained, they are compiled using msgfmt to create message catalogue files (*.mo files) for each language. Localized pages are served automatically based on the Accept-Language header sent by the user's web browser, or can be selected manually using language links in the page footer (this method also sets a cookie that serves future pages in that language). When the page is is sent to the visitor, the web server reads the relevant .mo file, looks up each string, and sends the translated text to the user instead of the English version.

Editing the website
Audacity keeps the web site's PHP source in SVN. So to edit the web site you need to check out a working copy of the source from the website directory in SVN. You can then edit the relevant page, and commit your changes back to SVN. At this point, nothing has changed on the web server, but the new version of the site has been stored in SVN.

Updating the web server from SVN
The copy of the web site that users actually see is located on SourceForge servers. In order for changes in SVN to become visible, they need to be transmitted there. '''This process currently only works on *nix-like systems (such as Linux, or Mac OS X if you use the command line tools). Users without a *nix workstation can perform the same steps on the Sourceforge Shell Service by logging in over SSH.'''

Prerequisites
To update the web site you will need all of the following installed on your system. Note that at this point you are not actually editing anything, so a remote shell login to a system (e.g. the Sourceforge Shell Service, which has all the required tools) is quite sufficient.
 * SVN (aka Subversion)
 * SSH: Needed as a transport layer for RSync.
 * RSync Used to transmit files to and from the SourceForge web server.
 * GNU Make: Used to automate commands in the translation process. Other makes may work but have not been tested.
 * GNU gettext tools: msgfmt and xgettext are needed to generate and compile translations.
 * Python: Audacity uses a customised message merging tool to deal with cases where a string is not translated but a very similar one is. This is written in Python, so your system needs to be able to run .py scripts.

You also need a working copy of the website SVN directory. If you are using Sourceforge Shell Service:
 * 1) cd 
 * 2) svn checkout https://audacity.googlecode.com/svn/website/trunk/htdocs --username 

Updating the site
This assumes you already have a working copy of the website SVN directory, and are in the top level htdocs folder.
 * 1) Set the SFUSER environment variable to your username on SourceForge, so the scripts know who to try and log in as. In the default bash shell, do something like export SFUSER=richardash1981 (yes, Sourceforge - as the web host is still there, even though the version control isn't)
 * 2) The next step may overwrite changes to your working copy. You should have already committed those changes to SVN, in which case that does not matter, but if you want to check what will happen, run make pretend-pull first.
 * 3) Run make pull to download files that aren't under SVN control from the web server into your working copy.
 * 4) Bring your SVN working copy up to date without extraneous directories by running svn up.
 * 5) Check the output to ensure that all version controlled files are in sync with SVN. In general if they are not, then the SVN version is the correct one, so conflicting or modified files can be reverted to the SVN repository version (use svn revert).  Files that are not in SVN should be left as-is, unless you know what you are doing / what they are.  They might be somebody's experimental files that the site relies on.
 * 6) If the .po files get into a mess, then you can delete all local changes to the .po files and get the SVN copies by doing make revert-po. This is just a short-cut to reverting them all with SVN.
 * 7) Run make pretend-push to see a list of altered files that will be uploaded to the SourceForge server (you will need the password for your SSH key (the one that communicates with Sourceforge web servers)). If this isn't what you expected, then check what is going now. No files are actually changed at this point.
 * 8) Run make. It will create first audacity_website.pot, then update community/potdates.inc with the date of the updated website.pot file on the Translators page, then update the .po files with the new strings, and finally compile all the .po files into .mo files. This can take a while and be quite CPU intensive.
 * 9) Run make pretend-push again. You should see that as well as the files you saw last time, locale/audacity_website.pot, community/potdates.inc and all the .mo files (in the individual language folders in locale) are scheduled to be uploaded to the server. No .po files will be uploaded because the rsync command ignores them (they are not needed on the web server).
 * 10) Finally run make push to upload the files.

Adding new web site translations
This consists of three operations - merging strings with the new .po file to bring it up to date, compiling it to generate .mo files for the web site, and committing the new translation to SVN.

Prerequisites
The same tools and checked out copy of the website SVN directory are needed as for updating the site from SVN above.

Updating a translation
This assumes you already have a working copy of the website SVN directory, and are in the top level htdocs folder.
 * 1) Set the SFUSER environment variable to your username on SourceForge, so the scripts know who to try and log in as.
 * 2) Run make pull to download files that aren't under SVN control from the web server into your working copy. This may overwrite changes to your working copy, but that doesn't matter provided that you have committed them to SVN.
 * 3) Bring your SVN working copy up to date without extraneous directories by running svn up.
 * 4) Check the output to ensure that all version controlled files are in sync with SVN. In general if they are not, then the SVN version is the correct one, so conflicting or modified files can be reverted. It is very likely that .po files will have conflicts, and the SVN version should be used. A quick hack to revert all .po files is to run make revert-po.
 * 5) Run make pretend-push to see a list of altered files that will be uploaded to the SourceForge server (you will need the password for your SSH key). This should be an empty list, as no files have changed. If not, then something isn't right. No files are actually changed at this point.
 * 6) Overwrite the relevant language translation with the new .po file you have been sent by the translator.
 * 7) Run make to generate new translations based on the new file. Any strings that have been altered since the last update by the translator will be merged into the .po file.
 * 8) Run make pretend-push again. You should see that the relevant language's .mo file, and possibly locale/audacity_website.pot, is scheduled to be uploaded to the server. No .po files will be uploaded because the rsync command ignores them (they are not needed on the web server).
 * 9) Run make push to upload the files.
 * 10) Commit the updated .po file to SVN in the normal way.

Line Endings and Cross-Platform Issues
The translation system will not work correctly if files with incorrect line endings end up on the (Linux) server where xgettext is run (by the Makefile) in order to create the .pot file. To prevent this, all files need to be stored in SVN with the svn:eol-style property set to native. This property must be set once when each file is committed to SVN. Note that if you do an SVN copy (preserving the history) then the properties are also kept. If you do an ordinary file copy, they aren't kept when you do an Add.

To locate files which have not been correctly set, use this BASH command: for file in $(find . -name '*.php' -or -name '*.inc'); do style=$(svn propget svn:eol-style ${file});if "${style}" != "native" ; then echo "${file}" ; fi ; done These files need to be converted to the native line ending for the platform being worked on, the property set and the lot committed. To check the commit you will find the --ignore-eol-style sub-option useful: svn di -x --ignore-eol-style

You can prevent future problems by enabling the "auto-props" feature in Subversion's configuration file. See http://www.mediawiki.org/wiki/Subversion/auto-props for help setting appropriate [auto-props] entries.

Hints for Tortoise SVN users on Windows: SVN properties can be added or edited by right-clicking over the file(s) > TortoiseSVN > Properties, or Properties then the "Subversion" tab of the Windows properties sheet. You can also edit the subversion config file on the "General" page of TortoiseSVN settings to set properties automatically on files and folders when they are added to the repository. To use native line endings on new files and folders, uncomment "enable-auto-props = yes" in config, then uncomment or add an appropriate entry (as above) in the [auto-props] section.

It's a lot easier if this is got right before files are added!