Summary of my GSoC project

The Google Summer of Code 2007 has finished. It's time to summarize the results.

Patches submitted to the ejabberd bug tracker

• XEP-0203: Delayed delivery
• XEP-0157: Contact Addresses for XMPP Services
• New ejabberd router for multicast packets
• mod_muc use XEP-0033 to reduce bandwidth consumption in message stanzas
• ejabberd_c2s use XEP-0033 to reduce bandwidth consumption in presence stanzas
• Service for XEP-0033: Extended Stanza Addressing
• XEP-0133: Service Administration
• Describe in developer doc how to receive answers to IQ in an ejabberd module

Protocols that I have (partially) read during my project

• XEP-0004: Data Forms
• XEP-0022: Message Events
• XEP-0030: Service Discovery
• XEP-0033: Extended Stanza Addressing
• XEP-0045: Multi-User Chat
• XEP-0050: Ad-Hoc Commands
• XEP-0068: Field Standardization for Data Forms
• XEP-0082: XMPP Date and Time Profiles
• XEP-0133: Service Administration
• XEP-0157: Contact Addresses for XMPP Services
• XEP-0160: Best Practices for Handling Offline Messages
• XEP-0202: Entity Time
• XEP-0203: Delayed delivery

Collateral tasks

• I reported to Peter Saint-Andre all the errors that I found in the XEPs.
• During my implementation of XEP-0033 I wrote several blog posts proposing changes in this protocol. Peter Saint-Andre will use those texts to update the protocol.
  
• Start to use Emacs to format Erlang code (emacs-mode) and commit to SVN repository (psvn).
• Start to read Joe Armstrong's new book: Programming Erlang - Software for a Concurrent World. Apply the new knowledge while programming in my GSoC tasks.
  
• Start to read GSoC gift book, Karl Fogel's Producing Open Source Software - How to Run a Successful Free Software Project. Apply the new knowledge in my ejabberd tasks.
• Continue my involvement in ejabberd as usual, which includes being active in ejabberd's forum, chatroom, mailing list and ejabberd-modules contribution SVN repository.
  
• Two travels, one of them international :)

Final GSoC project status

A week ago I posted my Almost final GSoC project status.

Since the previous status update I completed those tasks:

• Implement or update as much as possible XEP133 Service Administration in ejabberd.
• Prepare and submit patches to ejabberd bug tracker.

The tasks that I haven't completed and my plans to complete them are, in no special order:

• Perform code profiling to find bottlenecks and deficiencies in mod_multicast. Improve the code. - I'll focus in that topic from now on.
  
• Once I make all the possible optimizations: perform benchmarks to check mod_multicast's effect in CPU, RAM and traffic consumption.
  
• Wait for ejabberd code reviewers, in case I need to fix any problem in my code before they are applied to ejaabberd trunk.
• Discuss potential security and spam vulnerabilities (talk in JDEV and JADMIN mailint lists).
• Add XEP33 support to ejabberd's Pub/Sub and/or PEP service once their codebase is stable.
• Wait for Peter Saint-Andre's questions regarding his XEP-0033 update.
• September 7th: Upload final code to Google Summer of Code hosting.

The Google Summer of Code 2007 has finished, so those remaining tasks fall out of the scope of my GSoC project timeline. However, I consider them important for my own personal project timeline. So you can expect me to work on all of them at some time.

ejabberd gets XEP-0033: Extended Stanza Addressing

I consider finished my GSoC task of implementing XEP-0033: Extended Stanza Addressing in ejabberd.

The implementation is divided in several parts:

• New ejabberd module that provides a multicast service: mod_multicast.erl
• A small router for multicast packets: ejabberd_router_multicast.erl
  
• mod_muc uses the new multicast router for message stanzas
• ejabberd_c2s uses the new multicast router for presence stanzas

The largest part of the code is in the multicast service (mod_multicast).

Tomorrow is the GSoC pencils down deadline. This means that I will be evaluated only for the code that I wrote until today.

I expect all my code to be eventually included in ejabberd trunk. However, I'll propose that mod_multicast is disabled by default in the example configuration. At least in the first ejabberd release that includes that module.


Benchmark

I made some benchmarks using Jabsimul. The performance indexes that I could evaluate are only the %CPU and MB of RAM consumed by the ejabberd program. I created 900 accounts, and populated each one with around 40 roster items of type 'both'. Then, using Jabsimul each logged user changed its presence every few seconds.

With the patches and the multicast service enabled with small rosters and small chatrooms (less than 5 contacts or paticipants), there's a small increase in CPU consumption. With medium-size rosters (40 roster items), the CPU consumption triplicates in respect to the stock ejabberd trunk version.

Obviously, I don't consider acceptable when the CPU consumption is multiplied by 3 just because all the packets use XEP33 with 40 destinations. The bottleneck is mod_multicast.

However, this result does not surprise me at all. During my GSoC coding I only cared about optimization in the patches that will be commited to ejabberd trunk: ejabberd_c2s, mod_muc_room and ejabberd_router_multicast. I didn't care about code optimizations in mod_multicast. For me it was far more important the functionality correctness. Now that mod_multicast works correctly, I can concentrate in improving it without breaking its correctness.

This planning allowed me to do all the stuff that I planned for my GSoC project, and finish the summer with correct and working code. During the last week of August I plan to profile, reorganize and improve mod_multicast to reduce its computational consumption as much as possible.


Unexpected improvement

The funny thing is that my patches to ejabberd core with a disabled multicast service reduce slightly the CPU consumption compared to the stock ejabberd trunk version. This means that there is a possible optimization in ejabberd that does not deal with XEP33 at all. If properly investigated, this improvement could be included in ejabberd trunk and benefit all ejabberd deployments, not only the ones with multicast enabled.

Temporary Lists of Recipients - proposal for XEP33

When I started my Google Summer of Code project three months ago, Tobias Markmann pointed me to his Temporary Lists of Recipients proposal.

The purpose is to reduce even more the bandwidth consumption by sharing a common list of JIDs between the two entities which maintain a XEP33 communication.

The idea seems worth to be considered... once the current XEP33 is already implemented and deployed in the XMPP world. So I bookmark this proposal for future reference, and let's see what happens.

Multiple replyto and enforce all them in XEP33

Yesterday I was chatting about XEP33 with Elmex in the ejabberd chatroom. He point me to a strange topic in this protocol:
`There MAY be more than one replyto or replyroom on a stanza, in which case the reply stanza MUST be routed to all of the addresses.'
Here is the chatroom log.

What does that mean? If a client receives a message with extended stanza addresses, and 100 replyto or replyroom, and the user wants to answer, XEP33 forces the client to send the response to all 100 addresses. Why should we allow the sending entity to enforce the receiving entity to answer to all address, instead of giving him the power to answer only to some? In the email world is this enforcement also present?

I think this topic could be reconsidered for the next XEP33 version.

ejabberd gets XEP-0133: Service Administration

One of the minor tasks in my Google Summmer of Code project was to implement in ejabberd as many of the 31 commands described in XEP-0133: Service Administration as possible.

Aleksey Shchepin already implemented many commands in ejabberd more than 4 years ago. A year and a half ago Magnus Henoch updated them to use XEP-0050: Ad-Hoc Commands. So, I just had to update them a little to become XEP-0133 compliant:

• 23. Send Announcement to Online Users
• 24. Set Message of the Day
• 25. Edit Message of the Day
• 26. Delete Message of the Day

The commands that I implemented from scratch are:

• 1. Add User
• 2. Delete User
• 5. End User Session
• 6. Get User Password
• 7. Change User Password
• 9. Get User Last Login Time
• 10. Get User Statistics
• 13. Get Number of Registered Users
• 15. Get Number of Online Users
• 30. Restart Service
• 31. Shut Down Service

Other commands are not implemented, and I didn't add them because I consider ejabberd already provides other ways more suitable:

• 8. Get User Roster
• 18. Get List of Registered Users
• 20. Get List of Online Users
• 27. Set Welcome Message
• 28. Delete Welcome Message
• 29. Edit Admin List

And finally, I didn't implement those commands because they use features not available in ejabberd:

• 3. Disable User
• 4. Re-Enable User
• 11. Edit Blacklist
• 12. Edit Whitelist
• 14. Get Number of Disabled Users
• 16. Get Number of Active Users
• 17. Get Number of Idle Users
• 19. Get List of Disabled Users
• 21. Get List of Active Users
• 22. Get List of Idle Users

During this task, I found and reported some typo errors to the author of the XEP (Peter Saint-Andre).

Finally, I tested most commands with Tkabber SVN, Psi SVN and Gajim SVN. Sergei Golovan quickly fixed a small bug in Tkabber, and now all three clients work perfectly :)

I'm quite happy with the result, so I took this screenshot that depicts the impressive list of commands that allow an administrator to configure ejabberd just with a Jabber client:


Note that the commands are nested in the Service Discovery to allow the admin to find them easier.

The patch is available here. I hope it has quality enough to enter ejabberd trunk easily, so it is published in the next major ejabberd release.

Almost final GSoC project status

A month ago I posted my Midterm GSoC project status, and remaining work.

Since the previous status update I completed those tasks:

• Plan improvements in the current XEP33 definition of limits. I recently posted a summary: http://badlop.blogspot.com/2007/08/summary-of-xep33-addresses-limits.html
• Implement the improvements for limits in mod_multicast.
• Propose improvements for XEP33. The XEP33 wiki page lists the proposed changes.
• Test compatibility with other XEP33 existing implementations: Openfire, Psi, Tkabber. Results are reported in: XEP33 implementations: separate service or embedded support?
• Write documentation: configuration is explained in README.txt

The remaining tasks that I'm aware of, from now until the end of my GSoC project are:

• Implement or update as much as possible XEP133 Service Administration in ejabberd.
• Perform code profiling to find bottlenecks and deficiencies in mod_multicast. Improve the code.
  
• Perform benchmarks to check mod_multicast's effect in CPU, RAM and traffic consumption.
• Prepare and submit patches to ejabberd bug tracker.
• Upload final code to Google Summer of Code hosting.
• Wait for ejabberd code reviewers, in case I need to fix any problem in my code before commiting to ejabberd.
• Discuss potential security and spam vulnerabilities (talk in JDEV and JADMIN mailint lists).
• Add XEP33 support to ejabberd's Pub/Sub and/or PEP service if their codebase is stable at the time.