My C server so far

Source code can be found here: https://github.com/Lothy/Dionysus

I’ve only added it to github this early because I need some feedback – it doesn’t actually run yet (well, a client can successfully connect to it and reach the login screen, but that’s it).

I thought my current packet writing API was great, but didn’t consider setting the size of the packets after writing all of the appropriate data. Libevent API doesn’t support direct access to the underlying uchar array in a defined manner (if at all), so I need to redo it.
This means that I’m going to need to write data into uchar arrays that I control directly before sending the data using Libevent.

Fortunately Libevent supports sending the data in a uchar * by passing it in as a pointer, which means the data isn’t copied several times before actually being sent.
I can pass a pointer to a callback function as well which will be notified when the uchar * has been written.
One obvious solution to this problem is that I simply malloc two uchar * for each game tick.
I don’t really want to do this though, as I don’t believe mallocing/freeing potentially several KB per player (up to 2k players) per game tick (600ms) is ideal.

On the other hand, if you check out my bit_packer files (https://github.com/Lothy/Dionysus/blob/master/include/bit_packer.h and https://github.com/Lothy/Dionysus/blob/master/src/bit_packer.c) you’ll see that I have a bit_buffer struct.
I could alternatively have an array of a similar struct which contains an underlying uchar *.
I would then have a integer acting as a bitfield to indicate which elements in the array are still in use (marked with a 1 bit) and which are ‘free’ to use for a new write (0 bit).
The upside here is that I’m no longer constantly allocating and freeing a potentially large amount of memory.

The reason I thought of this is because of the Libevent API.
Specifically, the evbuffer_add_reference function (documentation here: http://www.monkey.org/~provos/libevent/doxygen-2.0.1/buffer_8h.html#8572c7c382489635be20d251039b8673).
When I pass a uchar * in, it will always read from index 0 – this rules out use of a true circular buffer.

Thoughts and opinions?
Also, for those who are C programmers (I’m guessing abraham, super_ and mopar plus maybe frank_ if he sees this), comment on the code.
Evil_ ran it through Valgrind a few days ago and it reported no memory leaks, so that’s a positive start in my opinion. :stuck_out_tongue:

With regards to ‘best practice’, I know that I need to move some variable declarations from header files to the respective source files. I’ll get to that, but I’m more interested in your thoughts on how I can handle the packet writing.

Second question: How does the game tick work client-side? What would happen if I changed the server’s tick from say 600ms to 100ms (and, as a consequence, changed the sending of position updates and so on only every 6th tick to ensure correct movement speed).

Thanks for the help guys.

“Cache IDs” is the 192-bit unique computer ID number, stored in random.dat. “CRC keys” i think are the checksums of each local resource descriptor table file – not positive as it’s been quite a long time since i’ve looked at a client. avoid “update keys” and implement a proper update server. in regards to smaller tick size, i’ve thought about it but i don’t think there is any real reason to.

Doesn’t seem like anyone can figure out the right way to do it though:
https://www.moparisthebest.com/smf/index.php/topic,427511.0.html

update keys are just crc and length of the indexs as I recall

I’m not at all concerned about providing a proper update server as part of my project.
Doing so automatically implies that I basically want to distribute their intellectual property, and that could make it liable (even just for removal from github).

Out of interest, does the client expect to receive update packets (ie, player/npc updates) back-to-back, or can the client receive a player update packet, and then perhaps a packet telling it to display a particular tab, and then the npc update packet?

I did some reading about the performance of malloc/free last night, and some implementations are really shit apparently. Someone mentioned that they’re avoided like the plague in real-time systems.
I was thinking that an 8KB or 16KB circular buffer would solve the problem – there’s no way that a cbuffer that size would wrap around before the contents at the start have been written.

You can provide your own implementations of malloc/free though can’t you? Perhaps using something nice like a heap? (or your circular buffer)

Well sure, but any allocation function that I was to write would leverage malloc anyway – the difference being that I would, on the first call to the function, just allocate a huge block of heap memory.
Knowing this, I think it’d be better to just have the circular buffer for each player as part of the application’s static memory.
Ignore that dumb moment above too – circular buffers would work fine, because a pointer doesn’t have to point to the start of a block of memory. :rolleyes: Too much java programming.

In terms of your question about the ticks Lothy, I’m reasonably sure that the official RuneScape server does all game logic processing in multiples of the 600ms tick so there shouldn’t be any need to lower it.

The update server I wrote for 317 is still up on GitHub and hasn’t been removed yet - https://github.com/apollo-rsps/jagcached.

The 317 update server in MoparScape 4 works perfectly as well, it’s the 508 no one seems to be able to get right.

Pushed some more changes to github just now before I go to bed.
I’ve implemented a circular buffer struct and associated functions which allow writing of primitive types to the circular buffer.
Having done some rudimentary testing, it seems to be bug free so far. It can wrap around right after an opcode write, right after a size write, or in the case of the size being 2 bytes, it can wrap around in the middle of the size write.

So yeah, seems to operate correctly.

I ported the bit packer code to the circular buffer files as well, and added wraparound functionality.
Just a question regarding that – with a circular buffer, obviously most of the buffer will contain non-zero values after the initial wraparound. When reading bits, does the client ignore any bits after the bits it expects to read? For example, if I write a packet that only uses two bits (ie, 11000000) in the last byte of the payload, will it matter if those remaining, unused, bits (such that 11000000 becomes something like 11001010) are set or not?

I didn’t have any problems implementing an update server for one of the newer clients (600+), and I don’t think 508 is very different.

[quote=“Metho D, post:11, topic:384832”][quote author=Moparisthebest link=topic=480598.msg3521016#msg3521016 date=1299402155]
The 317 update server in MoparScape 4 works perfectly as well, it’s the 508 no one seems to be able to get right.
[/quote]

I didn’t have any problems implementing an update server for one of the newer clients (600+), and I don’t think 508 is very different.[/quote]

Do you care to share it? Or at least have a look at mine and see where I went wrong?

[quote=“Moparisthebest, post:12, topic:384832”][quote author=Metho D link=topic=480598.msg3521284#msg3521284 date=1299426915]

I didn’t have any problems implementing an update server for one of the newer clients (600+), and I don’t think 508 is very different.
[/quote]

Do you care to share it? Or at least have a look at mine and see where I went wrong?[/quote]

I don’t know if I still have it, but I can take a look at yours if you provide the code.

[quote=“Metho D, post:13, topic:384832”][quote author=Moparisthebest link=topic=480598.msg3521696#msg3521696 date=1299445830]

Do you care to share it? Or at least have a look at mine and see where I went wrong?
[/quote]

I don’t know if I still have it, but I can take a look at yours if you provide the code.[/quote]

maybe you should read his thread.

At first glance, the only thing that looks problematic is your dumpFile method in the Main class (org.moparscape.cacheutils.v508.Main). It appears that you sometimes write more data to the file than you should. At the end of all files in the cache (with the exception of those with index = 255), a two-byte version value is appended that is used to see if a file is outdated and should be requested. The server doesn’t send this data with the file – the client appends it after the file has been received. If this is the problem, you should be able to fix it by modifying the loop on line 141 to go from [0…data.length - 2] instead of [0…data.length] if index != 255.

Hope that helps.

I’ll have a go at that later and report back, it would be great if it was that simple. :slight_smile:

On a happy note, I’ve just made some changes following frank_'s advice.
Previously I was using an if block to check for wraparound. Now the code uses the modulo operator for all wraparound detection (bitpacker as well).

Basically that part of the code should perform a lot better.
Hopefully I’ll have some time to make some more major progress on Wednesday and Thursday, as those two are uni-free (and with any luck I’ll actually be well rested lol).

Just finished writing a function to append player update blocks to the circular buffer.
If anyone wants to have a quick look over it to make sure I haven’t made any silly mistakes, feel free to: https://github.com/Lothy/Dionysus/blob/master/src/protocols/508.c

In particular, I need clarification that I have a) included all update blocks, b) have them all in the right order (although wrote a quick class as part of Blake’s server and used the comparator to sort a list of all player and npc update block implementations, and then printed them out in order).

Here’s the code if you CBF going to github for some reason :rolleyes:

int append_player_update_blocks(struct player *player,
        struct circular_buffer *cbuf)
{
    int i;
    int hp_ratio;
    int app_block_start_index;
    unsigned char app_block_size;
    struct item *equip_item;
    struct item_definition *item_def;

    if (player->update_flags & PUFLAG_SECONDARY_HIT_UPDATE_REQUIRED) {
        cwrite_byte_S(cbuf, player->secondary_damage.amount);
        cwrite_byte_A(cbuf, player->secondary_damage.type);
    }

    if (player->update_flags & PUFLAG_FACE_MOB_UPDATE_REQUIRED) {
        cwrite_short_B(cbuf, player->mob_to_face);
    }

    if (player->update_flags & PUFLAG_FORCED_CHAT_UPDATE_REQUIRED) {
        player->forced_chat[MAX_FORCED_CHAT_LEN - 1] = '\0';
        cwrite_string(cbuf, player->forced_chat);
    }

    if (player->update_flags & PUFLAG_GRAPHICS_UPDATE_REQUIRED) {
        cwrite_short_B(cbuf, player->graphic.graphic_id);
        cwrite_int_M(cbuf, player->graphic.height);
    }

    if (player->update_flags & PUFLAG_FACE_POSITION_UPDATE_REQUIRED) {
        cwrite_short_L(cbuf, (player->face_position_abs_x * 2) + 1);
        cwrite_short_BA(cbuf, (player->face_position_abs_y * 2) + 1);
    }

    if (player->update_flags & PUFLAG_CHAT_UPDATE_REQUIRED) {
        cwrite_short_BA(cbuf, player->chat.effects);
        cwrite_byte_C(cbuf, player->rights);
        cwrite_byte_C(cbuf, player->chat.message_len);
        player->chat.message[MAX_CHAT_LEN - 1] = '\0';
        for (i = 0; i < player->chat.message_len; ++i) {
            cwrite_byte(cbuf, player->chat.message[i]);
        }
    }

    if (player->update_flags & PUFLAG_ANIM_UPDATE_REQUIRED) {
        cwrite_short_B(cbuf, player->animation.anim_id);
        cwrite_byte_S(cbuf, player->animation.delay);
    }

    if (player->update_flags & PUFLAG_APPEARANCE_UPDATE_REQUIRED) {
        app_block_start_index = cwrite_byte(cbuf, 0); /* Placeholder */

        cwrite_byte(cbuf, player->appearance.gender);
        if ((player->appearance.gender & 0x2) == 2) {
            cwrite_byte(cbuf, 0);
            cwrite_byte(cbuf, 0);
        }
        cwrite_byte(cbuf, player->appearance.skull_icon);
        cwrite_byte(cbuf, player->appearance.head_icon);

        if (player->appearance.npc_id == -1) {
            for (i = 0; i < 4; ++i) {
                equip_item = &player->equipped_items[i];
                if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                    item_def = get_item_definition(equip_item->item_id);
                    cwrite_short_B(cbuf, 32768 + item_def->equip_id);
                }
                else {
                    cwrite_byte(cbuf, 0);
                }
            }

            /* Chest slot */
            equip_item = &player->equipped_items[EQUIPMENT_CHEST];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                item_def = get_item_definition(equip_item->item_id);
                cwrite_short_B(cbuf, 32768 + item_def->equip_id);
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_CHEST]);
            }

            /* Shield slot */
            equip_item = &player->equipped_items[EQUIPMENT_SHIELD];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                item_def = get_item_definition(equip_item->item_id);
                cwrite_short_B(cbuf, 32768 + item_def->equip_id);
            }
            else {
                cwrite_byte(cbuf, 0);
            }

            /* Arms slot */
            equip_item = &player->equipped_items[EQUIPMENT_CHEST];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                if (!is_full_body(equip_item->item_id)) {
                    cwrite_short_B(cbuf, 0x100
                            + player->appearance.looks[APPEARANCE_ARMS]);
                }
                else {
                    cwrite_byte(cbuf, 0);
                }
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_ARMS]);
            }

            /* Legs slot */
            equip_item = &player->equipped_items[EQUIPMENT_LEGS];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                item_def = get_item_definition(equip_item->item_id);
                cwrite_short_B(cbuf, 32768 + item_def->equip_id);
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_LEGS]);
            }

            /* Head slot */
            equip_item = &player->equipped_items[EQUIPMENT_HAT];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                if (is_full_helm(equip_item->item_id) || is_full_mask(
                        equip_item->item_id)) {
                    cwrite_byte(cbuf, 0);
                }
                else {
                    cwrite_short_B(cbuf, 0x100
                            + player->appearance.looks[APPEARANCE_HEAD]);
                }
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_HEAD]);
            }

            /* Hands slot */
            equip_item = &player->equipped_items[EQUIPMENT_HANDS];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                item_def = get_item_definition(equip_item->item_id);
                cwrite_short_B(cbuf, 32768 + item_def->equip_id);
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_HANDS]);
            }

            /* Feet slot */
            equip_item = &player->equipped_items[EQUIPMENT_FEET];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                item_def = get_item_definition(equip_item->item_id);
                cwrite_short_B(cbuf, 32768 + item_def->equip_id);
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_FEET]);
            }

            /* Beard slot */
            equip_item = &player->equipped_items[EQUIPMENT_HAT];
            if (equip_item->item_id != NO_ITEM_EQUIPPED) {
                if (is_full_mask(equip_item->item_id)) {
                    cwrite_byte(cbuf, 0);
                }
                else {
                    cwrite_short_B(cbuf, 0x100
                            + player->appearance.looks[APPEARANCE_BEARD]);
                }
            }
            else {
                cwrite_short_B(cbuf, 0x100
                        + player->appearance.looks[APPEARANCE_BEARD]);
            }
        }
        else {
            cwrite_short_B(cbuf, -1);
            cwrite_short_B(cbuf, player->appearance.npc_id);
        }

        for (i = 0; i < 7; ++i) {
            cwrite_byte(cbuf, player->appearance.colours[i]);
        }

        /* Animation indices */
        cwrite_short_B(cbuf, player->appearance.stand_anim);
        cwrite_short_B(cbuf, 0x337);
        cwrite_short_B(cbuf, player->appearance.walk_anim);
        cwrite_short_B(cbuf, 0x334);
        cwrite_short_B(cbuf, 0x335);
        cwrite_short_B(cbuf, 0x336);
        cwrite_short_B(cbuf, player->appearance.run_anim);

        cwrite_long_B(cbuf, player->username_hash);
        cwrite_byte(cbuf, player->combat_level);
        cwrite_short_B(cbuf, 0);

        /*
* We now need to write the size of the appearance
* update block into the placeholder that we
* set aside at the start.
* As we're using a circular buffer, we have to test
* for wraparound and deal with it.
*/
        if (cbuf->write_offset > app_block_start_index) { /* No wraparound */
            app_block_size = cbuf->write_offset - app_block_start_index;
            cbuf->buffer[app_block_start_index] = app_block_size;
        }
        else { /* Wraparound occurred */
            app_block_size = cbuf->write_offset;
            app_block_size += sizeof(cbuf->buffer) - app_block_start_index;
            cbuf->buffer[app_block_start_index] = app_block_size;
        }
    }

    if (player->update_flags & PUFLAG_PRIMARY_HIT_UPDATE_REQUIRED) {
        if (player->cur_skill_levels[SKILL_HITPOINTS] < 0) {
            player->cur_skill_levels[SKILL_HITPOINTS] = 0;
        }

        hp_ratio = (player->cur_skill_levels[SKILL_HITPOINTS] * 255)
                / player->max_skill_levels[SKILL_HITPOINTS];
        if (hp_ratio > 255) {
            hp_ratio = 255;
        }

        cwrite_byte_S(cbuf, player->primary_damage.amount);
        cwrite_byte_S(cbuf, player->primary_damage.type);
        cwrite_byte_S(cbuf, hp_ratio);
    }
    return 1;
}

int append_npc_update_blocks(struct npc *npc, struct circular_buffer *cbuf)
{
    if (npc->update_flags & NUFLAG_FACE_MOB_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_NPC_TRANSFORM_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_FORCED_CHAT_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_ANIM_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_GRAPHICS_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_SECONDARY_HIT_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_FACE_POSITION_UPDATE_REQUIRED) {

    }

    if (npc->update_flags & NUFLAG_PRIMARY_HIT_UPDATE_REQUIRED) {

    }
    return 1;
}

You’re missing the first player update block (indicated by 0x100). It’s used for agility and other purposes, and people generally call it “force movement”.

Everything else seems to be in order.

Okay, I’ll look into that later. That’s probably something that Blake is going to need to add to Osiris as well, seeing as the code above is completely based on his.

Thanks.