[dokuwiki] Re: DokuFS issue

Hi,

Stefan Waidele schrieb:
> I installed the developer-version (Sept 10th, 2008, 20:00) using the
> instructions from http://www.dokuwiki.org/devel:darcs
> 
> "darcs get --partial http://dev.splitbrain.org/darcs/dokuwiki";
> 
> I enabled XMLRPC in the configuration.

Okay, and that's part of your problem. During DokuWiki Hackdays some
functions of the XMLRPC-interface were changed and DokuFS is not yet
compatible with these changes. Nevertheless, you don't have to downgrade
(it should work with the latest stable release of DokuWiki) but you can
just upgrade DokuFS and enjoy yet more features and hopefully bugfixes.
E.g. media up- and downloads are possible now thanks to a new option
-media that switches DokuFS into "media-mode".

> On my desktop, I am using Ubuntu 7.10 - I had to install some
> Ruby-packet "sudo apt-get install libfusefs-ruby" and the
> dokufs.rb-skript worked.
> 
> Here's my experience:
> 
>   * I was able to create a page on my brand-new dev-wiki "start.dw" and
> it showed both in DokuFS and in the browser.
> 
>   * I edited the page in the browser, but the change did not show in the
> DokuFS.

This is because I do cache in all pages in DokuFS. Every five minutes I
pull the recent changes from DokuWiki and update all files that changed.
That means after a maximum of 5 minutes you should see the changes in
the DokuFS.

I might add another option for specifying this update interval or also a
cache timeout, that means that the page is deleted from the cache when
it's older than e.g. 2 minutes, but I don't know if that's a good
option. If you have any ideas for this, please let me know.

>   * After some reading and writing attempts to start.dw in DokuFS,
> dokufs.rb terminated with the following message:
> 
> ----8<--------------------------------------------------------------
> stw@fourgb:~$ ./dokufs.rb -user xxx -password yyy  -no-ssl -server
> stefan.waidele.info -path /wiki/lib/exe/xmlrpc.php dw
> /usr/lib/ruby/1.8/xmlrpc/client.rb:414:in `call': There are no changes
> in the specified timeframe (XMLRPC::FaultException)
>         from ./dokufs.rb:300:in `update'
>         from ./dokufs.rb:340
>         from ./dokufs.rb:338:in `initialize'
>         from ./dokufs.rb:338:in `new'
>         from ./dokufs.rb:338
> ----8<--------------------------------------------------------------

Interesting, yes, that's correct that you see this error because I don't
catch it. Didn't know the server sends such an error message but now I
see it and it's clear. DokuFS is still very bad in error checking. I've
added a check that should prevent this error.

>   * Any further attempt to mount the wiki failed like this:
> 
> ----8<--------------------------------------------------------------
> stw@fourgb:/SRV/home/stw$ ./dokufs.rb -user xxx -password yyy -no-ssl
> -server stefan.waidele.info -path /wiki/lib/exe/xmlrpc.php dw2
> ./dokufs.rb:109:in `initialize': private method `gsub' called for
> #<Hash:0xb7288874> (NoMethodError)
>         from ./dokufs.rb:108:in `each'
>         from ./dokufs.rb:108:in `initialize'
>         from ./dokufs.rb:335:in `new'
>         from ./dokufs.rb:335
> ----8<--------------------------------------------------------------
> 
> As you can see, I even created a new mountpoint.

And now that's the incompatibility I talked about. The data the server
sends has changed. Don't know why you haven't got it in the first place,
perhaps because your wiki was empty.

> I will reboot my workstation after sending this message to see if I can
> mount the wiki again after reboot and if I can reproduce the error.

You don't have to, as I said it's a compatibility problem.

> Let me know if I can supply any more information which could be usefull
> to improve this great project.

Thanks you!

I've attached a snapshot of my local development version. I guess it
still has some bugs (I still haven't found the time to write some more
and especially more sophisticated tests) but it works with the latest
version of DokuWiki, has the media-feature I already talked about
(although I am currently not using a cache for media-files which makes
the media-feature a bit slow when you use graphical programs that read
the files very often) and it should be able to handle acls much better
(that means e.g. you can't open pages that are read-only for writing, in
the previous version that was possible).

So the bugs you found shouldn't happen again in this version (hopefully ;)).

In the case you should detect any bugs please tell me (or write to the
list, but I doubt anybody but me will be helping and I don't know if
everybody is interested in these bugs as most of them only concern
DokuFS ;) ).

Greetings
Michael Hamann

#!/usr/bin/ruby
# DokuFS
# A Filesystem for accessing DokuWiki (version 2008-05-05 or above)
# on your local filesystem. More information can be found on 
# http://www.content-space.de/go/dokufs
#
# Copyright (C) 2008  Michael Hamann  michael <at> content-space.de

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# The idea of this program and some of the recursive functions here
# are adapted versions of MetaFS which was written by Greg Millam and
# distributed with FuseFS. Besides Ruby this FuseFS which can be
# obtained on http://rubyforge.org/projects/fusefs/ is the only
# dependency of DokuFS.

# TODO: recognize when save failed

require "cgi"
require "fusefs"
require "xmlrpc/client"

class StringCache < Hash
        def initialize (maxsize)
                super()
                @maxsize = maxsize
                @lru_keys = []
        end

        def clear
                super
                @lru_keys.clear
        end

        def []= (key, value)
                raise ArgumentError, "Value must be kind of String" unless 
value.kind_of?(String)
                remove_lru
                super
                touch key
        end

        def merge! (hash)
                hash.each { |k,v| self[k] = v }
        end

        def delete (key)
                value = super
                @lru_keys.delete key
                value
        end

        protected

        def touch (key)
                @lru_keys.delete key
                @lru_keys << key
        end

        def mem_size
                result = 0
                each_value do |v|
                        result += v.size
                end
                return result
        end

        def remove_lru
                while mem_size >= @maxsize
                        key = @lru_keys.delete_at 0
                        delete key
                end
        end
end

class DokuFS < FuseFS::FuseDir
        AUTH_NONE = 0
        AUTH_READ = 1
        AUTH_EDIT = 2
        AUTH_CREATE = 4
        AUTH_UPLOAD = 8
        AUTH_DELETE = 16
        AUTH_ADMIN = 255

        DEFAULT_OPTS = {
                :use_ssl => true,
                :path => "/lib/exe/xmlrpc.php",
                :host => "localhost"
        }

        def root?
                @is_root
        end

        def media?
                @is_media
        end

        def initialize(user_opts = nil)
                @pages = {}
                @subdirs = {}
                if ! user_opts.nil?
                        opts = DEFAULT_OPTS
                        opts.merge!(user_opts)
                        opts[:path] += 
"?u=#{CGI.escape(opts[:user])}&p=#{CGI.escape(opts[:password])}" if opts[:user] 
&& opts[:password]
                        @server = XMLRPC::Client.new3(opts)
                        @is_root = true
                        @is_media = opts[:media]
                        @last_update = Time.now.utc.to_i
                        unless self.media?
                                @cache = StringCache.new(1024*1024*5)
                                @server.call("wiki.getAllPages").each do |page|
                                        self.add(pagename_to_path(page['id']), 
page)
                                end
                        else
                                @server.call("wiki.getAttachments", "", 
{:recursive => true}).each do |media|
                                        self.add(pagename_to_path(media['id']), 
media)
                                end
                        end
                end
        end

        def add(path, data)
                base, rest = split_path(path)
                case
                when base.nil?
                        return false
                when rest.nil?
                        @pages[base] = data
                when @subdirs.has_key?(base)
                        @subdirs[base].add(rest, data)
                else
                        (@subdirs[base] = self.class.new).add(rest, data)
                end
        end

        def getdata(path)
                base, rest = split_path(path)
                case
                when base.nil?
                        false
                when rest.nil?
                        if @pages.has_key?(base)
                                return @pages[base]
                        else
                                return false
                        end
                when ! @subdirs.has_key?(base)
                        false
                else
                        @subdirs[base].getdata(rest)
                end
        end

        def contents(path)
                base, rest = split_path(path)
                case
                when base.nil?
                        (@pages.keys + @subdirs.keys).sort.uniq
                when ! @subdirs.has_key?(base)
                        nil
                when rest.nil?
                        @subdirs[base].contents('/')
                else
                        @subdirs[base].contents(rest)
                end
        end

        def size(path)
                if directory?(path)
                        return 4000
                else
                        if file?(path)
                                return getdata(path)['size']
                        end
                end
        end

        def directory?(path)
                base, rest = split_path(path)
                case
                when base.nil?
                        true
                when ! @subdirs.has_key?(base)
                        false
                when rest.nil?
                        true
                else
                        @subdirs[base].directory?(rest)
                end
        end

        def file?(path)
                if (self.root? && ! self.media?)
                        return true if @cache.has_key?(path_to_pagename(path))
                end
                base, rest = split_path(path)
                case
                when base.nil?
                        false
                when rest.nil?
                        @pages.has_key?(base)
                when ! @subdirs.has_key?(base)
                        false
                else
                        @subdirs[base].file?(rest)
                end
        end

        def can_write? path
                if file?(path)
                        perms = getdata(path)['perms']
                        if media?
                                return perms >= AUTH_DELETE
                        else
                                return perms >= AUTH_EDIT
                        end
                else
                        if media?
                                perms = @server.call('wiki.aclCheck', 
path_to_pagename(path)) 
                                return perms >= AUTH_UPLOAD
                        else
                                return false unless path =~ /\.dw\Z/
                                        perms = @server.call('wiki.aclCheck', 
path_to_pagename(path)) 
                                return perms >= AUTH_CREATE
                        end
                end
        end

        # mkdir
        def can_mkdir? path
                return true
        end

        def mkdir(path)
                base, rest = split_path(path)
                case
                when base.nil?
                        false
                when rest.nil?
                        @subdirs[base] = self.class.new
                        true
                when ! @subdirs.has_key?(base)
                        false
                else
                        @subdirs[base].mkdir(rest)
                end
        end

        # Delete a file
        def can_delete?(path)
                #return false unless Process.uid == FuseFS.reader_uid
                if file?(path)
                        perms = getdata(path)['perms']
                        if media?
                                return perms >= AUTH_DELETE
                        else
                                return perms >= AUTH_EDIT
                        end
                else
                        return false
                end
        end

        def remove_from_tree(path)
                base, rest = split_path(path)
                case
                when base.nil?
                        nil
                when rest.nil?
                        # Delete it.
                        @pages.delete(base)
                when ! @subdirs.has_key?(base)
                        nil
                else
                        @subdirs[base].remove_from_tree(rest)
                end
        end

        def delete(path)
                if media?
                        @server.call("wiki.deleteAttachment", 
path_to_pagename(path))
                else
                        @server.call("wiki.putPage", path_to_pagename(path), 
"", { "sum" => "deleted by DokuFS", "minor" => false })
                        @cache.delete(path_to_pagename(path))
                end
                self.remove_from_tree(path)
        end


        # Delete an existing directory.
        def can_rmdir?(path)
                #return false unless Process.uid == FuseFS.reader_uid
                base, rest = split_path(path)
                if base.nil?
                        @pages.empty?
                else
                        if @subdirs.has_key?(base)
                                if rest.nil?
                                        @subdirs[base].can_rmdir?("/")
                                else
                                        @subdirs[base].can_rmdir?(rest)
                                end
                        else
                                false
                        end
                end
        end
        def rmdir(path)
                base, rest = split_path(path)
                case
                when base.nil?
                        false
                when rest.nil?
                        @subdirs.delete(base)
                        true
                when ! @subdirs.has_key?(base)
                        false
                else
                        @subdirs[base].rmdir(rest)
                end
        end

        def read_file path
                pagename = path_to_pagename(path)
                if media?
                        
XMLRPC::Base64.decode(@server.call("wiki.getAttachment", pagename))
                else
                        @cache[pagename] ||= @server.call("wiki.getPage", 
pagename)
                end
        end

        def write_to (path, content)
                pagename = path_to_pagename(path)
                if media?
                        begin
                                encoded_content = XMLRPC::Base64.encode(content)
                                @server.call("wiki.putAttachment", pagename, 
encoded_content, {:overwrite => self.file?(path)})
                                data = {
                                        'id' => path_to_pagename(path),
                                        'size' => encoded_content.size,
                                        'perms' => 
@server.call('wiki.aclCheck', path_to_pagename(path)),
                                }
                                self.add(path, data)
                        rescue Exception => e
                                puts e.message
                        end
                else
                        message = { "sum" => "", "minor" => true }
                        plain_content = content
                        if content[0] == "%"[0]
                                content.sub!(/\A%\s?([^\n]+)\n?/m) do
                                        message["sum"] = $1
                                ""
                                end
                                message["minor"] = false
                        end
                        if content =~ /\A\s*\Z/m # when the page is empty, it 
is deleted
                                if message["sum"].empty?
                                        return false
                                end
                                self.remove_from_tree(path)
                        else
                                data = {
                                        'id' => path_to_pagename(path),
                                        'size' => plain_content.size,
                                        'perms' => 
@server.call('wiki.aclCheck', path_to_pagename(path)),
                                }
                                self.add(path, data)
                        end
                        @cache[pagename] = plain_content
                        @server.call("wiki.putPage", path_to_pagename(path), 
content, message)
                end
        end

        def update
                ltime = @last_update
                @last_update = Time.now.utc.to_i
    begin
      @server.call("wiki.getRecentChanges", ltime).each do |page|
        path = pagename_to_path(page["name"])
        if self.file?(path)
          @cache.delete(page["name"])
          self.remove_from_tree(path) if self.read_file(path).empty?
        else
          self.add(path, page)
        end
      end
    rescue XMLRPC::FaultException => e
    end
                return true
        end

        def path_to_pagename(path)
                path.sub(/\.dw\Z/, "").gsub("/", ":").reverse.chop.reverse
        end

        def pagename_to_path(id)
                if self.media?
                        return '/'+id.gsub(":", "/")
                else
                        return '/'+id.gsub(":", "/")+'.dw'
                end
        end
end

if (File.basename($0) == File.basename(__FILE__))
        Thread.abort_on_exception = true # for debugging...
        opts = {}
        begin
                arg = ARGV.shift
                case arg
                when "-user"
                        opts[:user] = ARGV.shift
                when "-password"
                        opts[:password] = ARGV.shift
                when "-server"
                        opts[:host] = ARGV.shift
                when "-path"
                        opts[:path] = ARGV.shift
                when "-no-ssl"
                        opts[:use_ssl] = false
                when "-media"
                        opts[:media] = true
                else
                        if ARGV.empty? && !arg.nil? && File.directory?(arg)
                                root = DokuFS.new(opts)
                                FuseFS.set_root(root)
                                FuseFS.mount_under(arg)
                                unless opts[:media]
                                        updater = Thread.new do
                                                sleep 5*60
                                                root.update
                                        end
                                end
                                FuseFS.run # This doesn't return until we're 
unmounted.
                                Thread.exit(updater) unless opts[:media]
                        else
                                puts <<-EOF
With DokuFS you can mount a DokuWiki under a path in your filesystem

All arguments except the path where to mount are optional, defaults are ssl, 
localhost as server and /lib/exe/xmlrpc.php as path. No authentication is 
default.

The -media-flag indicates if DokuFS should work in media-mode, that means it 
will display all media files instead of wiki-pages. Upload of media-files is 
possible.

Usage: dokufs.rb [-media] [-user your_username -password your_password] 
[-server your_server.com] [-path your/path/to/lib/exe/xmlrpc.php] [-no-ssl] 
path/where/to/mount/
                                EOF
        exit;
                        end
                end
        end while arg != nil
end

Other related posts: